Java

The Java compiler’s job is to check whether code follows the rules of the language. Trying to remove an element from a collection that can never contain it, or using the wrong letter in a date pattern, does not break those rules, so nothing stops the code from compiling. In short, any code that follows the rules compiles, even when it does not do what its author intended. What Is Error Prone? Error Prone is an open-source static analysis tool that Google built to fill this gap. Unlike most static analysis tools, it does not run as a separate step; it plugs into javac as a plugin. Using the information the compiler produces while analyzing the code, it looks for patterns that follow the rules of the language but are most likely wrong, and reports them as compiler warnings or errors. The idea is simple: a bug costs the least to fix when the code is written. Error Prone catches bugs during compilation, before they reach code review, CI, or production. Google has used the tool on its huge Java codebase for years, but its use is not limited to Google. NullAway, the null safety tool developed by Uber, runs as an Error Prone plugin. Spring Framework 7 checks its own codebase with NullAway during the build. Popular libraries such as Caffeine also use Error Prone in their builds. The examples in this article, which introduces you to Error Prone, are available in this repository. How It Differs From Other Static Analysis Tools The Java world has well-established tools such as SpotBugs, PMD, Checkstyle, and SonarQube. Three key features set Error Prone apart from them: It runs inside the compiler. It needs no separate build step or server. Because it is a plugin for javac, it works with any build tool that calls javac (Maven, Gradle, Bazel, Ant).It knows everything the compiler knows. The analysis runs on the AST that javac builds, after the type attribution and flow analysis phases. This gives Error Prone access to all symbol and type information. For example, it knows exactly what the generic type argument of an expression is, or which overload was chosen.It does not stop at a diagnosis; it suggests a fix. Most checks come with a suggested fix, which can be applied to the source code with a single command. How Does Error Prone Help Developers? The strength of Error Prone is that it can stop code that compiles but behaves incorrectly, right at compile time. Let’s take a closer look at the following example from the official documentation: Java Set<Short> s = new HashSet<>(); for (short i = 0; i < 100; i++) { s.add(i); s.remove(i - 1); } System.out.println(s.size()); This code seems to remove the previous element at each step, so you would expect it to print 1. It prints 100 instead, because the expression i - 1 is an int and is boxed to Integer. Since the Set<Short> contains no Integer, remove removes nothing. And because the remove(Object) signature accepts any type, the compiler does not complain either. Error Prone, however, refuses to compile this code: Plain Text [ERROR] ShortSet.java:[22,21] [CollectionIncompatibleType] Argument 'i - 1' should not be passed to this method; its type int is not compatible with its collection's type argument Short Now let’s look at the following code: Java private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("YYYY-MM-dd"); String reportName(LocalDate date) { return "report-" + FORMATTER.format(date) + ".csv"; } The code looks right when you read it, and it even works correctly for most of the year. However, Y in the pattern does not mean the calendar year; it means the week-based year. The calendar year is written with a lowercase y. When the last days of the year fall into the first week of the next year, the two values diverge. For example, for LocalDate.of(2025, 12, 29) the result is report-2026-12-29.csv, so the date jumps a year ahead. When the bug shows up also depends on the locale. The same code starts producing the wrong year on December 28 in the en-US locale, and on December 29 in the tr-TR locale. For the rest of the year the tests pass, and in code review the difference between YYYY and yyyy is easy to miss. Error Prone recognizes this pattern, stops the compilation and suggests the fix as well: Plain Text [ERROR] ReportDate.java:[16,83] [MisusedWeekYear] Use of "YYYY" (week year) in a date pattern without "ww" (week in year). You probably meant to use "yyyy" (year) instead. Did you mean 'private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd");'? Note that in both examples the code is valid Java and compiles without any problem. At best, your IDE shows a warning; the build does not break. Since the code can also pass the tests, the bug can live unnoticed for a long time. With Error Prone, you do not have to rely on anyone’s attention to catch these subtle bugs: the build fails, the error message explains the problem, and it often suggests the fix as well. Extending Error Prone Error Prone is not a closed set of rules. Where the built-in checks fall short, you can add your own rules on the same infrastructure they use. There are two ways to do this: Writing transformation rules with Refaster. Rules of the form “when you see this code, replace it with that” are written as plain Java templates.Writing your own BugChecker. Team-specific bug patterns are defined with the same API that Error Prone uses for its own checks. Refaster Refaster is a refactoring tool that is part of Error Prone, built for large-scale transformations across big codebases. You do not write the rule against an AST API, but in plain Java: “when you see this, replace it with that.” A Refaster rule is an ordinary Java class with at least two methods. @BeforeTemplate defines the pattern to look for, and @AfterTemplate defines the code to put in its place: Java class StringIsEmpty { @BeforeTemplate boolean before(String s) { return s.length() == 0; } @AfterTemplate boolean after(String s) { return s.isEmpty(); } } The template is matched with type information, not as text. That is why both name.length() == 0 and user.getEmail().length() == 0 are caught, while the same expression on a type other than String that also has a length() method is not. By writing more than one @BeforeTemplate, you can combine different patterns that lead to the same result in a single rule. Using a rule takes two steps: The rule class is compiled with the RefasterRuleCompiler plugin, which produces a .refaster file. The plugin lives in the error_prone_refaster artifact and is passed to the compiler with the -Xplugin:RefasterRuleCompiler --out rules.refaster argument.The generated .refaster file is passed to Error Prone with -XepPatchChecks:refaster:<file>. Error Prone writes the matches to a patch file, or applies them directly to the source code with -XepPatchLocation:IN_PLACE. Refaster is ideal for tasks such as migrating from one API to another, removing a deprecated method, or standardizing an idiom within a team. Picnic’s open-source Error Prone Support project also offers a way to run Refaster rules on every build, like a regular Error Prone check. Custom BugCheckers Error Prone’s built-in checks do not cover every rule. Alongside widely shared rules such as “sensitive data like national ID numbers, credit card numbers and email addresses must not be written to logs” or “do not use new BigDecimal(double) for monetary calculations", each team may also have its own rules, such as "this internal API must no longer be called". Error Prone lets you write these rules on the same infrastructure as its built-in checks. A custom check is a Java class that extends BugChecker. It declares which AST nodes it is interested in through the *TreeMatcher interfaces it implements. Error Prone calls the check for each such node, and the check returns a match and an optional fix. The following example implements the sensitive data rule. Although the rule is general, only the team knows which data is sensitive. That is why the team marks fields such as national ID numbers or email addresses with its own @SensitiveData annotation: Java public record Customer( String name, @SensitiveData String nationalId, @SensitiveData String email, @SensitiveData String cardNumber) {} The check reports an error when a log call is given a marked field, or an object whose toString() would print marked fields: Java @AutoService(BugChecker.class) @BugPattern(summary = "Sensitive data must not be written to logs", severity = ERROR) public final class SensitiveDataLogging extends BugChecker implements MethodInvocationTreeMatcher { private static final String SENSITIVE_DATA = "org.jugistanbul.SensitiveData"; private static final Matcher<ExpressionTree> LOGGER_CALL = instanceMethod() .onDescendantOfAny( "java.lang.System.Logger", "java.util.logging.Logger", "org.slf4j.Logger"); @Override public Description matchMethodInvocation(MethodInvocationTree tree, VisitorState state) { if (!LOGGER_CALL.matches(tree, state)) { return Description.NO_MATCH; } tree.getArguments().stream() .flatMap(argument -> stringifiedValues(argument, state)) // "..." + x → "...", x .filter(value -> isSensitive(value, state)) .forEach(value -> state.reportMatch( buildDescription(value) .setMessage("'%s' contains sensitive data and must not be written to logs" .formatted(state.getSourceForNode(value))) .build())); return Description.NO_MATCH; } private static boolean isSensitive(ExpressionTree value, VisitorState state) { // customer.email(): the field or accessor itself is annotated var symbol = ASTHelpers.getSymbol(value); if (symbol != null && ASTHelpers.hasAnnotation(symbol, SENSITIVE_DATA, state)) { return true; } // customer: the class, or one of its fields, is annotated, so toString() leaks it var type = ASTHelpers.getType(value); if (type == null || type.isPrimitive() || !(type.tsym instanceof ClassSymbol valueClass)) { return false; } return ASTHelpers.hasAnnotation(valueClass, SENSITIVE_DATA, state) || valueClass.getEnclosedElements().stream() .anyMatch(member -> member.getKind() == ElementKind.FIELD && ASTHelpers.hasAnnotation(member, SENSITIVE_DATA, state)); } // stringifiedValues(...) splits a string concatenation into its operands; see the repo } In the demo’s CheckoutService, the first line passes because it only logs the customer's name, while the other two do not compile: Java LOG.log(Level.INFO, "Order " + orderId + " placed by " + customer.name()); LOG.log(Level.DEBUG, "Order " + orderId + " placed by " + customer); LOG.log(Level.INFO, "Receipt for order " + orderId + " sent to " + customer.email()); Plain Text [ERROR] CheckoutService.java:[25,67] [SensitiveDataLogging] 'customer' contains sensitive data and must not be written to logs [ERROR] CheckoutService.java:[28,90] [SensitiveDataLogging] 'customer.email()' contains sensitive data and must not be written to logs The check looks at the expression passed to the logger, so a value first copied into a local variable (String email = customer.email();) is not tracked; following it would need data-flow analysis. This short class shows all the parts of a custom check: @BugPattern: Defines the rule's name, description, and severity (ERROR or WARNING). The rule can be turned off, or its severity changed, by its class name (-Xep:SensitiveDataLogging:WARN), and it can be suppressed with @SuppressWarnings.Matchers: Ready-made building blocks such as MethodMatchers express conditions like "any method of a System.Logger, java.util.logging or SLF4J logger" with type information, in a single expression.Type and symbol information: ASTHelpers gives access to the symbol and type of the logged value. Annotations are resolved by their fully qualified names, so an annotation with the same name in another package does not get mixed in. A text search cannot do this.@AutoService: Makes the check discoverable through ServiceLoader. Adding the jar that contains the checks to the annotation processor path is enough. This check only reports the error, but checks can also suggest fixes with SuggestedFix. BigDecimalDoubleConstructor in the repository is a short example that suggests BigDecimal.valueOf(double) instead of new BigDecimal(double). Checks are tested with CompilationTestHelper. // BUG: Diagnostic contains: comments in the test source mark the line where an error is expected. For checks that suggest fixes, BugCheckerRefactoringTestHelper compares the output of the fix with the expected source code; BigDecimalDoubleConstructorTest in the repository is an example. Running the Examples All the examples in this article are available in runnable form in the error-prone-in-action repository: ShortSet and ReportDate, which trigger the built-in checks, three custom checks with their tests, and the StringIsEmpty Refaster rule. The repository's README explains which command runs each example. Conclusion The Java compiler checks whether code follows the language rules, not what its author intended. Error Prone fills this gap from inside the compiler. Using javac’s type information, it catches code that compiles but is most likely wrong, and often suggests the fix as well. Where the built-in checks fall short, you can write transformation rules with Refaster and team-specific checks with the BugChecker API. In my view, this approach is more valuable today than ever. As AI coding assistants speed up the pace of writing code, tools that catch bugs at their cheapest moment, at compile time, are turning from a luxury into a necessity.

By Hüseyin Akdoğan CORE

Modern enterprise applications must process increasing data volumes while guaranteeing low latency, high availability, and predictable performance. While relational databases are essential for transactional workloads and complex queries, some applications do not fit the relational model. Systems that manage millions of user engagements, financial events, or IoT data often need a persistence solution built for distributed environments and horizontal scalability. In these cases, ScyllaDB affords a compelling option for enterprise architects. ScyllaDB is a high-performance, distributed NoSQL database based on the wide-column data model. Inspired by Apache Cassandra and compatible with its query language (CQL), ScyllaDB uses a shard-per-core architecture to maximize hardware efficiency and preserve consistent performance under intensive workloads. When should ScyllaDB be chosen over traditional relational databases or other NoSQL options? This article examines its column-family model, compares it with Cassandra, and shows how to integrate ScyllaDB into a Java enterprise application. Modern enterprise applications require more than basic data storage. They need high throughput, low latency, and horizontal scalability to manage complex workloads. ScyllaDB is a distributed, high-performance NoSQL database based on the wide-column data model, designed for applications that require predictable performance at scale. Companies such as Discord, Expedia, and Comcast use ScyllaDB in real-world scenarios, including messaging platforms, real-time applications, and high-volume data processing. ScyllaDB is especially well-suited for event-driven architectures, IoT platforms, financial systems, and applications with demanding read-and-write workloads. ScyllaDB uses a schema-based approach with Cassandra Query Language (CQL), which features SQL-like syntax, including statements such as CREATE TABLE, INSERT, and SELECT. Beyond primitive types, ScyllaDB supports collections like List, Set, and Map, as well as User-Defined Types (UDTs) for modeling complex data. This flexibility involves trade-offs: unlike relational databases, ScyllaDB uses query-driven data modeling, limits queries on arbitrary fields, and requires careful use of secondary indexes. This article will examine ScyllaDB’s column-family architecture, compare it with Apache Cassandra, review its capabilities and limitations, and show how to integrate ScyllaDB into enterprise Java applications. Setting Up ScyllaDB With Eclipse JNoSQL ScyllaDB supports deployment in cloud environments, on-premises data centers, and local development setups. Teams can select either self-managed or managed cloud services based on their infrastructure needs. In this tutorial, we will use a single ScyllaDB instance with Docker to focus on the data model and Java integration, avoiding the complexity of a distributed cluster. To start ScyllaDB locally, execute the following command: Shell docker run -d --name scylladb-instance -p 9042:9042 scylladb/scylla This command launches a ScyllaDB container and exposes port 9042, the default for Cassandra Query Language (CQL) connections. After initialization, the database is ready to accept application requests. Integrating ScyllaDB With Eclipse JNoSQL Next, determine how the Java application will connect to ScyllaDB. Starting with Eclipse JNoSQL 1.1.19, a dedicated driver provides native ScyllaDB support, enabling standard SQL operations alongside ScyllaDB-specific features. In this example, we use a Java application with Eclipse MicroProfile Config, CDI, and Jakarta JSON Processing. These technologies are common in MicroProfile and Jakarta EE environments, which helps with integration. First, add the ScyllaDB dependency to your Maven project: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-scylladb</artifactId> <version>${jnosql.version}</version> </dependency> Set the ${jnosql.version} property to Eclipse JNoSQL 1.1.19 or a compatible newer version. Configuring the Database Schema ScyllaDB uses a schema-based approach similar to Apache Cassandra. Developers define keyspaces, tables, column types, and user-defined types explicitly with CQL. In this example, we create a keyspace named developers, two tables for people and movies, and several user-defined types. Eclipse JNoSQL supports providing initialization queries through MicroProfile Config. Add the following properties to META-INF/microprofile-config.properties: Properties files jnosql.column.database=developers jnosql.scylladb.query.1=CREATE KEYSPACE IF NOT EXISTS developers WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor': 1}; jnosql.scylladb.query.2=CREATE COLUMNFAMILY IF NOT EXISTS developers.Person (id bigint PRIMARY KEY, name text, phones list<text>); jnosql.scylladb.query.3=CREATE TYPE IF NOT EXISTS developers.director (name text, movies set<text>); jnosql.scylladb.query.4=CREATE COLUMNFAMILY IF NOT EXISTS developers.Movie (name text PRIMARY KEY, age int, director FROZEN<director>); jnosql.scylladb.query.5=CREATE INDEX IF NOT EXISTS ageIndex ON developers.Movie(age); jnosql.scylladb.query.6=CREATE TYPE IF NOT EXISTS developers.money (currency text, amount decimal); jnosql.scylladb.query.7=CREATE TYPE IF NOT EXISTS developers.headquarter (city text, country text); This configuration highlights multiple key points. CQL is similar to SQL, using statements likeCREATE, explicit types, and primary keys. ScyllaDB supports collections such as list and set, allowing you to represent multiple values without extra tables. User-defined types (UDTs) group related attributes into structured objects, such as the director type. The Movie table uses FROZEN<director>. In CQL, FROZEN stores the UDT as a single value, preventing independent modification of individual fields. We also created a secondary index on the age column. Although ScyllaDB supports secondary indexes, their effectiveness depends on the workload, cardinality, and query types. Do not treat them as equivalent to relational database indexes or as a substitute for proper partition-key design. Defining database schemas through application configuration works for tutorials but isn't recommended for production. Manage schema evolution separately using version-controlled migration tools such as Liquibase, which supports ScyllaDB via its Cassandra extension. This approach separates application deployment from schema changes and improves control over database updates. Mapping Entities and User-Defined Types With the database schema defined, we can create the Java models. We start with Movie, representing a ScyllaDB entity, and Director, representing a user-defined type. Java @Entity("Movie") public class Movie { @Id("name") private String name; @Column private Integer age; @Column(udt = "director") private Director director; } public class Director { @Column private String name; @Column private Set<String> movies; } The Movie entity uses the standard Jakarta NoSQL mapping approach with @Entity, @Id, and @Column. The director attribute maps to a ScyllaDB user-defined type via @Column(udt = "director"). During serialization and deserialization, Eclipse JNoSQL uses the ScyllaDB driver to convert between the Java object and its CQL UDT representation. A UDT is similar to a structured JSON object that groups related fields, but it's a strongly typed CQL value defined in the schema. The Set<String> field in Director demonstrates ScyllaDB’s native support for collections, allowing you to model a director’s movie titles without a separate relational table. Next, we will define the Person entity: Java @Entity("Person") public class Person { @Id("id") private long id; @Column private String name; @Column private List<String> phones; } The phones attribute demonstrates another collection type ScyllaDB supports: list<text>. Eclipse JNoSQL maps Java List<String> directly to the corresponding CQL collection without extra converters or custom serialization. These examples show how ScyllaDB combines a schema-based model with support for structured types and collections. Eclipse JNoSQL offers familiar mapping annotations, permitting applications to use ScyllaDB’s native features. However, object mapping convenience does not replace the need for query-driven data modeling. In distributed databases like ScyllaDB, partition keys, access patterns, and data distribution remain essential architectural considerations. Running the Application: Jakarta NoSQL and Jakarta Data Once you configure the database and map entities, the next step is to interact with ScyllaDB from Java. Eclipse JNoSQL provides two complementary approaches: Jakarta NoSQL, a template-based API for NoSQL databases, and Jakarta Data, which uses the Repository pattern. Both integrate with CDI, letting developers inject persistence components into applications running on either Java SE with CDI or Jakarta EE application servers. Jakarta NoSQL: The Template Approach The first approach uses Jakarta NoSQL with the ColumnTemplate interface, offering a consistent API for wide-column databases. Developers can insert, update, delete, and retrieve entities without interacting directly with the database driver. Java @Inject ColumnTemplate template; For example, you can create a Person entity, insert it, and retrieve it using a fluent query API: Java Person person = Person.builder() .withId(1L) .withName("Ada Lovelace") .withPhones(List.of("234", "432")) .build(); template.insert(person); Optional<Person> result = template.select(Person.class) .where("id").eq(1L) .singleResult(); The entity mapping automatically handles the Java List<String> and its corresponding CQL collection, so you don't need extra serialization code to persist phone numbers. The Template approach is useful when developers need explicit control over persistence operations, want to build queries programmatically, or prefer a consistent API across NoSQL implementations. Jakarta Data: The Repository Approach The second approach uses Jakarta Data and applies the Repository pattern to simplify persistence. Developers define an interface, and Eclipse JNoSQL supplies the implementation. Consider our PersonRepository: Java @Repository public interface PersonRepository extends CrudRepository<Person, Long> { } This interface provides standard CRUD operations, such as save, findById, and deleteById. The repository can be injected with CDI and used directly in the application: Java @Inject PersonRepository repository; repository.save(person); Optional<Person> result = repository.findById(1L); Jakarta Data also supports query methods derived from method names, allowing developers to express queries without writing implementations. MovieRepository demonstrates this capability and exposes ScyllaDB-specific functionality through its dedicated interface: Java @Repository public interface MovieRepository extends ScyllaDBRepository<Movie, String> { List<Movie> findByAge(Integer age); @CQL("SELECT * FROM developers.Movie") List<Movie> findAllQuery(); } The findByAge method demonstrates query derivation, while the @CQL annotation lets you execute native Cassandra Query Language statements. In this example, the age column has a secondary index, so you can query movies by age without using the partition key. However, query derivation does not remove ScyllaDB’s querying restrictions. Repository methods must match queries supported by the schema and indexes. Retrieving all records is useful for demonstrations but should be used cautiously in production with large datasets. Beyond Standard CRUD Operations While Jakarta NoSQL and Jakarta Data simplify common persistence operations, ScyllaDB integration extends beyond standard repositories and templates. Eclipse JNoSQL offers ScyllaDBTemplate, a specialized implementation that exposes database-specific features while maintaining the familiar Java programming model. With this integration, developers can: Control consistency levels: Specify Cassandra-compatible consistency levels, such as ConsistencyLevel.ONE, to balance consistency, latency, and availability.Execute native CQL: Run database-specific queries directly when standard abstractions are insufficient, including those using ScyllaDB-specific features.Work with structured data: Map collections such as List and Set, as well as user-defined types, without manually converting Java objects to CQL values.Choose the right abstraction: Use the vendor-neutral ColumnTemplate for general persistence, or ScyllaDBTemplate for additional database control. For instance, the specialized template supports an operation such as: Java scyllaDBTemplate.save(person, ConsistencyLevel.ONE); A key architectural feature of Eclipse JNoSQL is that abstraction does not mean losing access to database-specific capabilities. Developers can use standardized APIs for common operations and still access native functionality when needed. Jakarta NoSQL and Jakarta Data complement each other, offering different ways to organize persistence logic. The Template approach provides explicit, flexible interaction, while the Repository approach emphasizes declarative data access and separation of concerns. Both benefit from CDI integration, allowing you to use the same persistence components in standalone Java applications, microservices, or Jakarta EE environments. The choice depends on the application’s architectural requirements, but the underlying ScyllaDB capabilities and data-modeling constraints remain the same. Conclusion ScyllaDB is a strong choice for enterprise architects, especially for applications that demand high throughput, consistent latency, and horizontal scalability. Its schema-based wide-column model, support for collections and user-defined types, and compatibility with Cassandra Query Language provide a familiar and robust solution for distributed data persistence. However, to make informed architectural decisions, it's important to understand its query limitations, secondary indexes, and data-modeling trade-offs. Source: https://github.com/JNOSQL/demos-se#scylla.

By Otavio Santana CORE

What does it take to build and sustain an open-source project for more than a decade? For Shai Almog, creator of Codename One, it takes a willingness to challenge conventional approaches to software development and a continued commitment to the developers who rely on the work. I recently caught up with Shai to discuss the inspiration behind Codename One, the realities of maintaining an open-source project, and his perspective on how the Java ecosystem has evolved over the years. Check it out below! 1. What is Codename One, and what originally inspired you to create it? Codename One is what we always wanted Java to be: write once, run anywhere, without accepting the lowest common denominator. It’s what you would get if Spring Boot, Flutter, Go, and Node.js had a baby that inherited the best qualities of each. You get Java’s mature ecosystem, Flutter-style control over the UI/portability, Go-like native binaries and footprint, and the full-stack client/server connection people associate with Node.js. With Codename One, one Java codebase can target iOS, Android, macOS, Windows on x64 and ARM64, Linux on x64 and ARM64 with either glibc or musl, and even the browser. A small self-contained desktop GUI application can be around 4–5 MB, with no JVM to install. Underneath, Codename One uses different pipelines for different targets. On our ahead-of-time compiled platforms, ParparVM analyzes the reachable Java bytecode, removes unused code, translates what remains to C, and hands the result to the platform’s native compiler. Codename One also provides the portability libraries, UI toolkit, tests, theming, and build system needed to handle the ugly platform-specific details. We don’t pretend every platform is identical. Our builder architecture assembles the native pieces each application needs, so developers can use platform-specific capabilities without fragmenting the entire codebase. The original inspiration was frustration with Java’s unfulfilled promise. Java gave us portability, but conventional Java application deployment became increasingly heavy. We wanted the language and ecosystem without requiring a large runtime or giving up native performance, platform access, or control of the UI. Now we’re taking that architecture to the backend. The goal is to bring the startup time, footprint, and native compilation work we developed for constrained devices into serverless and microservice workloads. I want Java developers to get the full-stack productivity they expect from Spring Boot, with deployment economics closer to Go and Node.js. 2. Codename One has been around for more than a decade. What has kept you motivated? The community is the biggest reason. People have built businesses and products on Codename One. When someone has trusted your work with something that important, walking away isn’t an option. For a long time, competing in this market was brutal. Our competitors were not normal startups. They were Google, Facebook, Microsoft, and companies with teams larger than our entire community. There was a limit to how much a small team could build, test, document, and support. Agentic development changed that equation. It lets us scale engineering work in a way that simply wasn’t possible a few years ago. We now maintain an enormous cross-platform test matrix and can move at a pace that is competitive with much larger organizations. On most feature dimensions, I would argue that we are already ahead of the big-name alternatives. We still have gaps. We’re a small team, and pretending otherwise would be ridiculous. But our size is also an advantage. The person reading a bug report can be the same person who changes the compiler. We can respond to developers in hours where a large company may need three teams, a quarterly roadmap, and permission from a product manager. The other reason is simpler: we love Java, and we use the competing tools. Every time we do, we run into one of the compromises Codename One was created to remove. 3. What’s one of the biggest misconceptions developers have about maintaining an open source project? That maintaining open source is mostly about accepting code contributions. A pull request looks like free work from the outside. For a maintainer, it can be the beginning of a large amount of work. Is the code correct? Does it meet the project’s standards? Does it work on every target? Who maintains it five years later? How do you tell a volunteer who spent a week on something that the design is wrong? How many rounds of changes can you reasonably ask them to make? We recently stopped accepting outside pull requests. AI wasn’t the reason. If anything, AI improved the baseline quality of many contributions. It also made some of them enormous, because generating another thousand lines became almost free while reviewing those thousand lines did not. The real problem was validation. Codename One currently has more than 60 GitHub Actions workflow files. A small framework change can affect Android, iOS, JavaScript, desktop, bytecode translation, native compilation, and screenshot baselines. That matrix represents roughly $15,000 worth of monthly CI capacity, which GitHub currently provides to us. At that point, validating and repairing an outside patch often took longer than implementing the confirmed change ourselves. So today we ask people to contribute through reproducible issues, benchmark counterexamples, design discussions, testing, and documentation feedback. Those contributions are often more valuable than code because they expose problems we cannot reproduce in our own environments. Also sorry for all the GitHub Actions downtime ;-) 4. You’ve watched Java and the broader development ecosystem evolve considerably. What changes have surprised you the most? The biggest surprise is that Java effectively surrendered the idea of being small. GraalVM is a remarkable piece of engineering, so we aren’t literally alone (although it's a very different project that hasn't really taken the same direction and hasn't taken it as far). But mainstream Java spent years treating bigness as evidence of seriousness: larger runtimes, larger frameworks, more dynamic machinery, longer warmups, and more memory. The industry began treating that weight as inevitable. I'm not against comprehensive/large; Codename One itself is HUGE... But not in runtime, just in API and scope. Most successful newer platforms moved in the opposite direction. Go made self-contained native binaries normal. Node.js made application deployment almost trivial. Mobile and serverless made startup time and memory consumption economically important again. Java largely responded by optimizing large systems rather than questioning why every system needed to be large. Modules are a good example. They help organize and encapsulate a large dependency graph. They do not eliminate that graph or turn it into a small, closed-world native program. Codename One puts a real Java GUI application or full server into a roughly 5 MB binary. What surprises me is not that this works. We have spent years proving that it works. What surprises me is how lonely that position still is. Java could be one of the best languages for small native applications and low-cost serverless workloads. Most of the ecosystem simply stopped trying to make that happen and let Go eat our lunch. 5. After spending so much of your career in technology, what do you enjoy doing when you completely step away from a screen? Completely step away from a screen? I’m probably the wrong person to ask. I just got back from Thailand with my kids. We had a wonderful time. I also had my laptop with me and was doing pull requests on the beach. This is what I enjoy doing. I don’t have an unwind button. Vacations often leave me more exhausted than work because sitting still doesn’t recharge me. If I have an idea and I can build it, fix it, or argue about it with another developer, that is how I unwind. Explore more of Shai's content here.

If you've been following the database world lately, you'll have noticed DuckDB turning up in some unexpected places. One of the more interesting examples is pg_duckdb, a Postgres extension that embeds DuckDB directly into the Postgres process, letting analytical queries run through DuckDB's engine without leaving the database. It's a clever idea: keep your data where it is, but bring a faster analytical engine. That got us wondering: could we do something similar with Neo4j? The answer is not in the same way. Postgres has a mature extension architecture that allows foreign query engines to be embedded at a low level. Neo4j doesn't expose that kind of hook. But Neo4j does let you write custom procedures in Java that run inside the Neo4j JVM and are callable directly from Cypher. Also, DuckDB ships an official JDBC driver that bundles its native binary and loads it via JNI. Put those two things together and you've got something that feels surprisingly close to the pg_duckdb experience: DuckDB running in-process, queries fired from Cypher, results flowing back into a graph. That's what we built. In this article, we'll walk through the whole thing from scratch. The full source code is available on GitHub. Prerequisites Before starting, we'll need the following: JDK 21 – check with java -versionMaven – check with mvn -versionNeo4j Desktop – with a local database instance created and ready to start Note that this approach requires a self-managed Neo4j install. AuraDB doesn't support custom plugins, so Neo4j Desktop is the right choice here. Version Summary For reference, here are all the component versions used in this article: ComponentVersionOpenJDK21.0.12.1Maven3.9.6Neo4j Desktop2.2.1Neo4j2026.07.0Neo4j Procedure API5.26.0DuckDB JDBC1.1.3JUnit Jupiter5.10.2maven-compiler-plugin3.11.0maven-shade-plugin3.5.2maven-surefire-plugin3.5.4 Getting the Code The full source code is on GitHub. To get just this project: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set duckdb-procedure mv duckdb-procedure ../duckdb-procedure cd ../duckdb-procedure/neo4j-duckdb-procedure Or follow along from scratch using the steps below. How It Works The architecture is straightforward. We write two Java classes and package them as a fat jar: DuckDBConnection manages a singleton DuckDB connection that lives for the lifetime of the Neo4j process. It uses double-checked locking to keep things thread-safe and reconnects automatically if the connection is ever closed.DuckDBProcedure is the Neo4j procedure itself. It accepts a SQL string from Cypher, runs it against the shared DuckDB connection and streams the results back as rows, each one a Map<String, Object> that Cypher can destructure with dot notation. DuckDB runs in-memory by default, which means it starts clean every time Neo4j restarts. That's fine for analytics against external files, and we'll discuss file-backed persistence in the article Summary section. Project Setup Start by creating the project directory: Shell mkdir -p neo4j-duckdb-procedure/src/main/java/com/example/duckdb cd neo4j-duckdb-procedure Verify the structure: Shell find . -type d Now create pom.xml in the project root. XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>neo4j-duckdb-procedure</artifactId> <version>1.0.0</version> <packaging>jar</packaging> <properties> <java.version>21</java.version> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>5.26.0</neo4j.version> <duckdb.version>1.1.3</duckdb.version> </properties> <dependencies> <!-- Neo4j procedure API, provided by Neo4j at runtime --> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <!-- DuckDB JDBC driver, bundled with native binary --> <dependency> <groupId>org.duckdb</groupId> <artifactId>duckdb_jdbc</artifactId> <version>${duckdb.version}</version> </dependency> <!-- Test dependencies --> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <version>5.10.2</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.5.4</version> </plugin> <!-- Fat jar: bundles DuckDB JDBC into the plugin jar --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.2</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> The Neo4j dependency is provided scope — it won't be bundled into the jar since Neo4j supplies it at runtime. The DuckDB JDBC driver is included at compile scope so it gets bundled along with its native binary. The JUnit dependency is test scope so it's excluded from the fat jar. The maven-shade-plugin builds the fat jar at package time. Confirm Maven can resolve the dependencies: Shell mvn dependency:resolve The Java Classes Create DuckDBConnection.java at src/main/java/com/example/duckdb/: Java package com.example.duckdb; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; public class DuckDBConnection { private static volatile Connection instance; private DuckDBConnection() {} public static Connection getInstance() throws SQLException { if (instance == null || instance.isClosed()) { synchronized (DuckDBConnection.class) { if (instance == null || instance.isClosed()) { instance = DriverManager.getConnection("jdbc:duckdb:"); } } } return instance; } } The volatile keyword ensures the instance is visible across threads without full synchronization on every read. The jdbc:duckdb: connection string with no path gives us an in-memory DuckDB instance. Now create DuckDBProcedure.java in the same package: Java package com.example.duckdb; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.Procedure; import java.sql.Connection; import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.Statement; import java.sql.Types; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.stream.Stream; public class DuckDBProcedure { public static class QueryResult { public Map<String, Object> row; public QueryResult(Map<String, Object> row) { this.row = row; } } @Procedure(name = "duckdb.query") @Description("Run a DuckDB SQL query and return results as rows") public Stream<QueryResult> query(@Name("sql") String sql) throws Exception { List<QueryResult> results = new ArrayList<>(); synchronized (DuckDBConnection.class) { Connection conn = DuckDBConnection.getInstance(); try (Statement stmt = conn.createStatement()) { boolean hasResultSet = stmt.execute(sql); if (hasResultSet) { try (ResultSet rs = stmt.getResultSet()) { ResultSetMetaData meta = rs.getMetaData(); int columnCount = meta.getColumnCount(); while (rs.next()) { Map<String, Object> row = new HashMap<>(); for (int i = 1; i <= columnCount; i++) { String colName = meta.getColumnName(i); int colType = meta.getColumnType(i); String typeName = meta.getColumnTypeName(i); Object value = mapValue(rs, i, colType, typeName); row.put(colName, value); } results.add(new QueryResult(row)); } } } else { Map<String, Object> status = new HashMap<>(); status.put("status", "OK"); status.put("updateCount", (long) stmt.getUpdateCount()); results.add(new QueryResult(status)); } } } return results.stream(); } private Object mapValue(ResultSet rs, int index, int sqlType, String typeName) throws Exception { if (typeName != null && typeName.equalsIgnoreCase("HUGEINT")) { long val = rs.getLong(index); return rs.wasNull() ? null : val; } switch (sqlType) { case Types.TINYINT: case Types.SMALLINT: case Types.INTEGER: { long val = (long) rs.getInt(index); return rs.wasNull() ? null : val; } case Types.BIGINT: { long val = rs.getLong(index); return rs.wasNull() ? null : val; } case Types.FLOAT: case Types.REAL: case Types.DOUBLE: case Types.NUMERIC: case Types.DECIMAL: { double val = rs.getDouble(index); return rs.wasNull() ? null : val; } case Types.BOOLEAN: case Types.BIT: { boolean val = rs.getBoolean(index); return rs.wasNull() ? null : val; } case Types.NULL: return null; default: return rs.getString(index); } } } A few things worth noting about DuckDBProcedure: We use stmt.execute(sql) rather than executeQuery() because DuckDB write operations like COPY don't return a ResultSet. The boolean return value tells us which path to take.The mapValue method handles type mapping explicitly. DuckDB's HUGEINT type – used for aggregated sums — has no standard java.sql.Types mapping, so we check the type name directly. Every numeric read is followed by rs.wasNull() to avoid returning 0 for a null column.We synchronize on DuckDBConnection.class to keep concurrent Cypher calls thread-safe against the shared connection. Writing the Tests We tried neo4j-harness first, which spins up a lightweight embedded Neo4j instance in JUnit — the same approach used in the Neo4j Procedure Template. In our environment, we found that the embedded server took longer than expected to become ready during the test run, which led us to a simpler alternative. The fix is to skip neo4j-harness entirely and test the DuckDB logic directly via JDBC. Since DuckDBProcedure is a thin wrapper around a JDBC connection, we can exercise the same type mapping, null handling, and file reading logic without Neo4j in the loop at all. The tests run in under 4 seconds and are fully self-contained. First, add the test directory structure and a src/test/resources folder to hold the test data files: Shell mkdir -p src/test/java/com/example/duckdb mkdir -p src/test/resources The test data files cover all the types and edge cases we want to verify: Plain Text src/test/resources/ ├── test.csv ├── test_nulls.csv ├── test_floats.csv ├── test_booleans.csv ├── test_timezones.csv └── test.json Maven copies everything in src/test/resources to target/test-classes automatically, so the test class can resolve file paths via the classloader without hardcoding any absolute paths. The test class at src/test/java/com/example/duckdb/DuckDBProcedureTest.java covers 12 cases: Range queriesCSV readingParquet write and readJSON readingAggregation returning numeric values rather than stringsNull handlingFloatsBooleansEmpty result setsBad SQL error surfacingIn-memory table persistence across callsDuckDB-side JOIN between two files Each test mirrors something we verified manually in Neo4j. File paths are resolved portably so the tests work regardless of where the project is cloned: Java private String resourcePath(String filename) { return getClass().getClassLoader().getResource(filename).getPath(); } The Parquet test is the one exception — since DuckDB generates the Parquet file rather than reading a pre-existing one, it writes to a JUnit @TempDir that's cleaned up automatically after the test run. Run the full suite with: Shell mvn test All 12 tests should pass in a few seconds. Running mvn package will also run the tests before building the fat jar, so any regression in the DuckDB logic is caught before the jar is deployed. Building and Installing Build the fat jar: Shell mvn package -DskipTests Any warning about MANIFEST.MF from the shade plugin is harmless. The jar lands at target/neo4j-duckdb-procedure-1.0.0.jar. Copy it to your Neo4j Desktop plugins folder. You can find the path in Neo4j Desktop under the three-dot menu for your database instance, then Open Folder > Plugins. Then copy the jar: Shell cp target/neo4j-duckdb-procedure-1.0.0.jar "/path/to/your/neo4j/plugins/" Open neo4j.conf in the conf folder alongside plugins and add: Plain Text dbms.security.procedures.unrestricted=duckdb.* Restart the database. Confirm the procedure is registered using Query from the left-hand navigation pane: Cypher SHOW PROCEDURES YIELD name WHERE name STARTS WITH 'duckdb' RETURN name You should see duckdb.query. If the procedure doesn't appear after restarting, also add the following in neo4j.conf: Plain Text dbms.security.procedures.allowlist=duckdb.* Some Neo4j configurations require procedures to be explicitly allowlisted before they're loaded at startup. Putting It to Work The test data files used here are also available in src/test/resources in the project repository. Let's start simple. DuckDB has a built-in range function we can use without any external data: Cypher CALL duckdb.query("SELECT range AS num FROM range(1, 6)") YIELD row RETURN row.num AS num Five rows should be returned, confirming that DuckDB is running inside Neo4j. Note that DuckDB's range() function is exclusive of the upper bound, so range(1, 6) returns 1 through 5. Now let's read a CSV file. Create a test file at /tmp/test.csv: Plain Text city,country,population London,UK,9648110 Paris,France,2161000 Berlin,Germany,3645000 Tokyo,Japan,13960000 New York,USA,8336817 Query it from Cypher: Cypher CALL duckdb.query("SELECT city, country, population FROM read_csv_auto('/tmp/test.csv') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population DuckDB reads and parses the CSV; Neo4j receives the result. We can also write the CSV out as a Parquet file: Cypher CALL duckdb.query("COPY (SELECT * FROM read_csv_auto('/tmp/test.csv')) TO '/tmp/test.parquet' (FORMAT PARQUET)") YIELD row RETURN row And read it back: Cypher CALL duckdb.query("SELECT city, country, population FROM read_parquet('/tmp/test.parquet') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population DuckDB handles JSON just as easily. Create /tmp/test.json: JSON [ {"city": "London", "country": "UK", "population": 9648110}, {"city": "Paris", "country": "France", "population": 2161000}, {"city": "Berlin", "country": "Germany", "population": 3645000}, {"city": "Tokyo", "country": "Japan", "population": 13960000}, {"city": "New York", "country": "USA", "population": 8336817} ] And read it: Cypher CALL duckdb.query("SELECT city, country, population FROM read_json_auto('/tmp/test.json') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population We can also join two files entirely inside DuckDB before the result reaches Cypher. Create /tmp/test_timezones.csv: Plain Text city,timezone London,GMT Paris,CET Berlin,CET Tokyo,JST New York,EST Then join it against the population data: Cypher CALL duckdb.query("SELECT p.city, p.population, t.timezone FROM read_csv_auto('/tmp/test.csv') p JOIN read_csv_auto('/tmp/test_timezones.csv') t ON p.city = t.city ORDER BY p.population DESC") YIELD row RETURN row.city AS city, row.population AS population, row.timezone AS timezone The join happens entirely inside DuckDB. Neo4j just receives the final result set. Now for the most interesting demo: combining DuckDB analytics with Neo4j graph data in a single query. First, create some graph nodes: Cypher UNWIND [ {city: "London", continent: "Europe"}, {city: "Paris", continent: "Europe"}, {city: "Berlin", continent: "Europe"}, {city: "Tokyo", continent: "Asia"}, {city: "New York", continent: "North America"} ] AS data MERGE (c:City {name: data.city}) SET c.continent = data.continent Now join DuckDB file data with Neo4j graph data by city name: Cypher CALL duckdb.query("SELECT city, population FROM read_parquet('/tmp/test.parquet')") YIELD row MATCH (c:City {name: row.city}) RETURN c.name AS city, c.continent AS continent, row.population AS population ORDER BY row.population DESC And aggregate by continent — population figures from DuckDB, continent groupings from the graph: Cypher CALL duckdb.query("SELECT city, population FROM read_parquet('/tmp/test.parquet')") YIELD row MATCH (c:City {name: row.city}) RETURN c.continent AS continent, count(c) AS city_count, sum(row.population) AS total_population ORDER BY total_population DESC That's two engines contributing to a single result: DuckDB handles the file analytics, Neo4j handles the graph traversal and aggregation. What We Learned Building this wasn't entirely smooth, and the bumps are worth documenting. XML comments in a Maven POM can't contain double dashes anywhere except at the closing -->. That's a quirk of XML parsing that's easy to miss.DuckDB's HUGEINT type – used for the result of aggregation functions like SUM – has no corresponding entry in the standard java.sql.Types constants. Our initial type mapping switch statement silently fell through to the string default, returning numeric values as strings. The fix is to check ResultSetMetaData.getColumnTypeName() by name before entering the switch.Write operations like COPY can't go through executeQuery() since they don't return a ResultSet. Switching to stmt.execute() and checking the boolean return value handles both read and write paths cleanly.Null numeric values in JDBC return 0 from getInt() and getLong(), not null. You must call rs.wasNull() immediately after each numeric read to distinguish a genuine zero from a missing value. Comparing Approaches It's also worth noting that APOC (Awesome Procedures on Cypher) Extended's apoc.load.jdbc procedure supports DuckDB via its JDBC driver and apoc.jdbc.analytics goes further by letting you export graph data into a temporary DuckDB table for analytical queries. These are complementary approaches worth exploring alongside the custom procedure built here. The following table compares the three ways to use DuckDB with Neo4j, each with a different integration style. ApproachHow It WorksDirectionPersistent StateRequiresCustom procedure (this article)DuckDB runs in-process inside the Neo4j JVM as a singleton connectionFiles and SQL into Neo4jYes, across Cypher callsCustom Java pluginAPOC apoc.load.jdbcDuckDB treated as an external JDBC data source; driver jar dropped into plugins folderFiles and SQL into Neo4jNoAPOC Extended + DuckDB JDBC jarAPOC apoc.jdbc.analyticsGraph data exported into a temporary DuckDB table; analytical SQL runs against itGraph data into DuckDB for analyticsNoAPOC Extended + DuckDB JDBC jar Summary We started with a simple question: could DuckDB be embedded in Neo4j the way pg_duckdb embeds it in Postgres? The answer is not natively, but closer than you might expect. With two Java classes, a fat jar, and a few lines of Neo4j configuration, we built a working duckdb.query procedure that runs DuckDB SQL directly from Cypher. Along the way we handled CSV, Parquet and JSON files, DuckDB-side joins, cross-engine aggregations with Neo4j graph data and a handful of type mapping edge cases that the 12-test suite now catches automatically. There's plenty of room to grow this further. File-backed DuckDB, a proper connection pool, cleaner error handling, and possibly a maintained open-source plugin are all natural next steps. But the foundation is there. What we've built here is a proof of concept — it works, and the tests confirm it, but anyone taking it further should treat robustness, connection pooling, and error handling as open items before putting it near production data. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Virtual threads let us keep ordinary blocking code while many more requests wait concurrently. Under load, however, that brings three critical questions into focus: How much work do we admit? Cheap threads can still build an expensive queue.What belongs inside a lock? A sleeping lock holder still makes everyone else wait.Who stops abandoned work? A caller's timeout does not cancel its task. The examples below isolate these questions with separate before-and-after programs. Each program is complete: save its code block under the filename shown and run the accompanying command, without arguments. All you need is JDK 25. They use its compact source-file syntax, so no class declaration, build tool, external library, or preview flag is required. First: More Overlapping Waits Imagine 1,000 tasks that each wait for 100 ms, with no dependency limiting their concurrency: Execution modelSimultaneous waitsIdeal elapsed timePool of 50 platform threads502,000 msOne virtual thread per task, all waits overlapping1,000100 msVirtual threads limited to 50 concurrent waits502,000 ms These are arithmetic estimates, ignoring startup and scheduling overhead. With 50 workers, the waits take twenty batches. Allow all 1,000 waits to overlap, and they take one batch. Cap virtual-thread concurrency at 50 and the twenty-batch result returns. Virtual threads let more independent waits overlap. They do not execute CPU instructions faster. A virtual thread runs on a platform thread called a carrier. Many blocking JDK operations, including socket I/O and sleep, let it unmount while waiting. Its stack and referenced objects remain on the heap, and the carrier can run something else. (The M:N scheduling model: Many virtual threads map to a smaller pool of OS carrier threads) The scheduler's default target parallelism is the number of processors available to the JVM. Some blocking operations still occupy a carrier; native and foreign-function calls can pin virtual threads even on JDK 25. Waiting uses fewer OS threads, but it still costs memory and resources. 1. Bound Admission: Eight Slots Cannot Serve Everybody at Once Imagine a dependency with eight connections. Each call takes 80 ms. Its ideal capacity is 8 / 0.08 = 100 calls per second. Virtual threads can wait cheaply for a connection, but they cannot create another one. Send a burst of 100 requests. First, accept everything. Then, repeat with an admission limit of 32: up to eight using the dependency and another 24 waiting for it. First, accept every request. Save as UnboundedAdmission.java: Java import java.time.Duration; import java.time.Instant; import java.util.ArrayList; import java.util.concurrent.*; void main() throws Exception { var connections = new Semaphore(8); var results = new ArrayList<Future<?>>(); Instant now = Instant.now(); try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { for (int i = 0; i < 100; i++) { results.add(executor.submit(() -> { connections.acquire(); try { Thread.sleep(Duration.ofMillis(80)); // the dependency call } finally { connections.release(); } return null; })); } } // wait for all work to finish for (var result : results) result.get(); IO.println("completed=%d rejected=0 elapsed_ms=%d".formatted( results.size(), Duration.between(now, Instant.now()).toMillis())); } Run it: Java java UnboundedAdmission.java Every request waits until a connection becomes available. Now add an admission semaphore so excess requests are rejected immediately. Save as BoundedAdmission.java: Java import java.time.Duration; import java.time.Instant; import java.util.ArrayList; import java.util.concurrent.*; void main() throws Exception { var connections = new Semaphore(8); var admission = new Semaphore(32); var results = new ArrayList<Future<Boolean>>(); Instant now = Instant.now(); try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { for (int i = 0; i < 100; i++) { results.add(executor.submit(() -> { if (!admission.tryAcquire()) return false; try { connections.acquire(); try { Thread.sleep(Duration.ofMillis(80)); // the dependency call } finally { connections.release(); } return true; } finally { admission.release(); } })); } } // wait for accepted work to finish int completed = 0; for (var result : results) if (result.get()) completed++; IO.println("completed=%d rejected=%d elapsed_ms=%d".formatted( completed, 100 - completed, Duration.between(now, Instant.now()).toMillis())); } Run it: Java java BoundedAdmission.java One run on Oracle GraalVM 25.0.2 produced: AdmissionCompletedRejectedElapsed timeUnlimited10001,062 msLimit of 323268330 ms There are two different limits here. connections.acquire() waits for one of eight service slots. admission.tryAcquire() admits the request immediately or returns false, recording a rejection. Its permit stays held while the request waits for a connection and makes the call. Without admission control, all 100 requests complete in thirteen batches, taking roughly a second. With the gate, a quick burst typically admits 32 and rejects 68; four batches take roughly 320 ms. The bounded run finishes sooner because it rejects excess work, preventing memory exhaustion. A 10 KB per-thread cache across 100,000 virtual threads is still 1 GB of heap space. The takeaway: Reject before adding work to the queue you need to bound. 2. Shorten the Lock: Protect the Update Now use forty tasks. Each waits 80 ms and increments a shared completion counter. There is no connection limit here: the waits are independent, so the lock is the only bottleneck. First, put both the wait and the counter update inside the lock. Save as WideLock.java: Java import java.time.Duration; import java.time.Instant; import java.util.ArrayList; import java.util.concurrent.*; final Object lock = new Object(); int completed; void main() throws Exception { Instant now = Instant.now(); try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { var tasks = new ArrayList<Future<?>>(); for (int i = 0; i < 40; i++) { tasks.add(executor.submit(() -> { synchronized (lock) { Thread.sleep(Duration.ofMillis(80)); completed++; } return null; })); } for (var task : tasks) task.get(); } if (completed != 40) throw new AssertionError("Lost an update"); IO.println("completed=%d elapsed_ms=%d".formatted( completed, Duration.between(now, Instant.now()).toMillis())); } Run it: Java java WideLock.java Now move the wait outside the lock, keeping the counter update protected. Save as NarrowLock.java: Java import java.time.Duration; import java.time.Instant; import java.util.ArrayList; import java.util.concurrent.*; final Object lock = new Object(); int completed; void main() throws Exception { Instant now = Instant.now(); try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { var tasks = new ArrayList<Future<?>>(); for (int i = 0; i < 40; i++) { tasks.add(executor.submit(() -> { Thread.sleep(Duration.ofMillis(80)); synchronized (lock) { completed++; } return null; })); } for (var task : tasks) task.get(); } if (completed != 40) throw new AssertionError("Lost an update"); IO.println("completed=%d elapsed_ms=%d".formatted( completed, Duration.between(now, Instant.now()).toMillis())); } Run it: Java java NarrowLock.java On the same JDK, the two runs produced: Lock scopeCompletedElapsed timeWait and update inside403,255 msOnly the update inside4087 ms With the wait inside the lock, forty waits of 80 ms take at least 3,200 ms. Making the other thirty-nine threads cheap to park does not let them pass the lock holder. With the wait outside, those waits overlap. On JDK 25, synchronized is perfectly suitable for this demonstration, as JDK 24 removed monitor-related virtual-thread pinning. The issue here isn't platform pinning; it's basic thread serialization. The rule is to identify the state transition that needs protection, then keep independent slow work outside of it. 3. Cancel the Work: A Timeout Only Ends the Wait Use one task that takes 1,800 ms. Its caller is willing to wait only 200 ms. First, let the caller time out without canceling the task. Save as TimeoutOnly.java: Java import java.time.Duration; import java.time.Instant; import java.util.concurrent.*; void main() throws Exception { Instant now = Instant.now(); try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { var task = executor.submit(() -> { try { Thread.sleep(Duration.ofMillis(1_800)); IO.println("Work completed"); return "done"; } finally { IO.println("Cleanup ran"); } }); try { task.get(200, TimeUnit.MILLISECONDS); } catch (TimeoutException e) { IO.println("Caller timed out"); } } // close() still waits for the task to exit IO.println("scope_exited_ms=%d".formatted( Duration.between(now, Instant.now()).toMillis())); } Run it: Java java TimeoutOnly.java Now add task.cancel(true) to the timeout handler. Save as TimeoutWithCancellation.java: Java import java.time.Duration; import java.time.Instant; import java.util.concurrent.*; void main() throws Exception { Instant now = Instant.now(); try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { var task = executor.submit(() -> { try { Thread.sleep(Duration.ofMillis(1_800)); IO.println("Work completed"); return "done"; } finally { IO.println("Cleanup ran"); } }); try { task.get(200, TimeUnit.MILLISECONDS); } catch (TimeoutException e) { IO.println("Caller timed out"); task.cancel(true); // <--- Explicit cancellation } } // close() waits for the task to exit, even after cancellation IO.println("scope_exited_ms=%d".formatted( Duration.between(now, Instant.now()).toMillis())); } Run it: Java java TimeoutWithCancellation.java The observed outcomes were: ExamplePrinted Work completed?Printed Cleanup ran?Scope exited afterTimeout aloneYesYes1,805 msTimeout plus cancellationNoYes207 ms Without cancellation, get() throws after 200 ms, but the task keeps sleeping. It eventually completes its work 1.6 seconds later. If this abandoned work was holding a database connection or an admission permit, that resource would be leaked for the duration of the sleep. Cancellation is cooperative. Detect the timeout, request cancellation, and wait for the cleanup. All three parts matter. A Teaser: The Structured Concurrency Solution In the final example above, you'll notice the parent thread is forced to micromanage the task's lifecycle. It waits, catches the TimeoutException, explicitly calls cancel(true), and crosses its fingers that the cleanup runs. But what if we could invert this control? With JDK's preview StructuredTaskScope API, we can tie the lifetime of the threads to the lexical scope itself, making cancellation an automatic architectural guarantee rather than manual boilerplate: I will be exploring how StructuredTaskScope eliminates resource leaks and abandoned work entirely in my next article. Stay tuned! Summary Take these three questions back to your services when migrating to virtual threads: Where does waiting accumulate? Bound admission before that queue grows.What serializes the work? Keep independent waits outside state-protecting locks.Who owns unfinished work? Connect deadlines to cooperative cancellation and cleanup. The programs here use sleeps so each effect is visible, but the exact same questions apply when your wait is for a database, a socket, or a partner API.

By A N M Bazlur Rahman CORE

Microservices do not eliminate transactions. They force architects to decide where a transaction should end. That decision is harder than it sounds. A business operation that was once handled inside a single database transaction may cross several services after decomposition. An order can involve inventory, payment, pricing, customer limits, audit history, and notification. The architecture may divide these responsibilities cleanly, but the business still sees one operation: place the order. This is where many distributed systems become fragile. Teams draw service boundaries first and discuss consistency later. By then, business rules already cross databases, synchronous calls have formed long dependency chains, and eventual consistency has become a general excuse for states nobody can clearly explain. The better approach is to begin with business invariants. First decide which facts must remain true together. Then place transaction and service boundaries around those facts. Technology comes after that decision. Start With the Invariant, Not the Service An invariant is a business rule that must remain true whenever an operation is considered complete. It is stronger than ordinary validation. For example: A confirmed reservation must own the seat assigned to it.A financial transfer must preserve the accounting relationship between debit and credit entries.An accepted command must not create the same business effect twice.An order must not be marked as confirmed if the required payment authorization failed. These rules define the real consistency boundary. If two pieces of state must change together for an invariant to remain true, the safest design is usually to keep them inside one local transaction. Splitting them across services does not remove the rule. It converts the rule into a distributed coordination problem. That conversion may sometimes be necessary, but it should be a conscious tradeoff. A service boundary is not free merely because it looks clean on an architecture diagram. Three Different Consistency Problems Architects often discuss consistency as though the system has only two choices: strong consistency or eventual consistency. In practice, a distributed workflow usually contains three different boundaries. Figure 1. Three consistency boundaries: local atomicity, durable event publication, and eventual convergence across independently committed services. 1. The Atomic Core The atomic core contains state changes that must succeed or fail together. A local ACID transaction is still the simplest and strongest tool for protecting this boundary. Consider an order service that accepts a command, creates the order, records its initial status, and saves a durable event describing the accepted order. If losing that event would leave downstream systems permanently unaware of the order, the order record and the event must be committed together. This does not require a distributed transaction. Both records can live in the same database and participate in the same local transaction. 2. The Durable Handoff Once the atomic state is committed, the system must communicate the result to other components. This is a durability problem, not just a messaging problem. Publishing directly to a broker inside application logic creates a dangerous gap. The database commit may succeed while the publish fails. Publishing first creates the opposite problem: another service may receive an event for state that never committed. The transactional outbox pattern closes this gap by storing the business change and the outgoing event in the same local transaction. A separate publisher later sends the event to the broker. The handoff is durable because the intent to publish survives process crashes and temporary broker failures. 3. The Convergent Workflow After the event crosses a service boundary, the workflow is no longer atomic. Each service commits its own state at a different time. The architecture must therefore define how the system reaches a valid final state. This is where eventual consistency belongs. It should describe an explicit convergence process, not an undefined period in which anything may happen. The design must answer: Which intermediate states are valid?How long may the workflow remain incomplete?What happens when a step fails permanently?Can a completed action be compensated?Which component owns the workflow state?How can support teams and auditors determine what happened? If these questions have no clear answers, the design is not eventually consistent. It is merely incomplete. A Java Boundary Using a Transactional Outbox The following simplified Spring example keeps the order and its outgoing event inside one transaction: Java @Service public class OrderApplicationService { private final OrderRepository orderRepository; private final OutboxRepository outboxRepository; public OrderApplicationService( OrderRepository orderRepository, OutboxRepository outboxRepository) { this.orderRepository = orderRepository; this.outboxRepository = outboxRepository; } @Transactional public OrderId placeOrder(PlaceOrder command) { Order order = Order.create( command.customerId(), command.items()); orderRepository.save(order); OrderPlaced payload = new OrderPlaced( order.id().value(), order.customerId().value(), order.total(), order.version()); outboxRepository.save( OutboxMessage.pending( UUID.randomUUID(), "OrderPlaced", order.id().value(), payload)); return order.id(); } } The important detail is not the framework annotation. It is the placement of the boundary. The order and the outbox message are committed in the same database transaction. Either both exist, or neither exists. An outbox publisher can read pending messages and publish them asynchronously. If it crashes after publishing but before marking the message as sent, it may publish the same event again. That is acceptable only when consumers can recognize repeated business effects. The outbox solves atomic handoff, but it does not remove the need for idempotent processing. This distinction matters. Each pattern solves a specific failure window. Combining patterns without understanding their boundaries creates complexity without correctness. Model the Workflow as State, Not as a Call Chain A common mistake is to implement a distributed transaction as a chain of synchronous calls: Plain Text Order → Inventory → Payment → Shipping → Notification The chain looks simple when every call succeeds. Under failure, however, it behaves like a distributed transaction without the guarantees of one. Suppose inventory is reserved and the payment provider completes the authorization, but its response times out before reaching the order service. The order service cannot safely assume that payment failed. Retrying may create another authorization, while immediately starting compensation may cancel an order that was already paid. These situations should be modeled as durable business states rather than hidden inside request-handling logic. The normal path moves from PENDING to INVENTORY_RESERVED, then PAYMENT_PENDING, and finally CONFIRMED. A definitive payment rejection moves the workflow to PAYMENT_FAILED, followed by inventory release and cancellation. A timeout requires a different path. The workflow moves to PAYMENT_STATUS_UNKNOWN because the outcome is uncertain. A reconciliation step must query the payment provider or process a later payment event before deciding whether to confirm the order or continue with compensation. Figure 2. A durable order workflow distinguishes definitive payment failure from an unknown payment result that requires reconciliation. Each transition should have a durable cause, a stable business identifier, an expected previous state, and an idempotency rule. The component that owns the workflow must record progress before initiating the next step. With this design, the workflow can resume after a process crash because its progress exists as data rather than only inside a thread or request stack. It also gives architects, developers, and support teams an authoritative explanation of what happened and what the system should do next. This does not mean every workflow requires a large orchestration platform. A state table and a small coordinator may be sufficient. What matters is that intermediate states, failure paths, and recovery decisions are explicit. When Eventual Consistency Is Safe Eventual consistency is a good choice when temporary disagreement is acceptable, and convergence can be demonstrated. Typical examples include: Updating a search index after the source record commitsSending confirmation notificationsRefreshing analytics or reporting projectionsUpdating recommendations or cached viewsPropagating noncritical profile changes These operations can usually lag without violating the core business transaction. The decision becomes more difficult around inventory, credit limits, payment authorization, and regulatory records. These domains often contain rules that cannot tolerate arbitrary intermediate states. They may still require distributed workflows, but the workflow must use reservation, versioning, expiration, reconciliation, or another explicit control. For example, inventory does not need to be permanently deducted at the moment an order starts. It can be reserved for a limited period. The reservation becomes a first-class domain concept with an owner, an expiration time, and a release rule. That model is more honest than pretending an asynchronous deduction behaves like a local transaction. Database per Service Is a Guideline, Not the Starting Point Database ownership is important because shared schemas create coupling. However, “database per service” should not be applied before understanding the domain. If two pieces of data participate in the same invariant and change together frequently, separating them can create more coupling rather than less. The services may become independently deployable on paper while requiring constant runtime coordination in production. Sometimes the correct first architecture is a modular monolith with strong internal boundaries and one transactional database. Services can be extracted later when their business boundaries, scaling needs, and consistency requirements are understood. This is not a rejection of microservices. It is recognition that distribution should solve a real problem. Network boundaries are useful for independent ownership, deployment, scaling, security, and fault isolation. They are costly when placed inside an invariant that demands constant coordination. A Practical Decision Test Before splitting an operation across services, architects should ask the following questions: QuestionArchitectural implicationMust these changes always be visible together?Keep them in one local transaction when possible.Can the business tolerate an incomplete state?If yes, define the allowed state and its maximum duration.Can an action be safely retried?Require a stable business identifier and an idempotency rule.Can a completed action be reversed?Define compensation as a separate business operation.Who owns progress across services?Use an explicit coordinator or clearly documented choreography.How will the system detect stalled work?Persist deadlines and support reconciliation.What evidence explains the final state?Record transitions, causes, versions, and correlation identifiers. These questions should be answered before choosing a broker, workflow engine, or framework. Infrastructure can transport messages and execute retries, but it cannot decide which temporary states the business permits. Common Boundary Mistakes Several design mistakes appear repeatedly in distributed Java systems. Splitting First and Coordinating Later Teams decompose a system around nouns or technical layers, then discover that every important operation crosses several services. The resulting architecture has many services but few independent business capabilities. Calling Everything Eventually Consistent Eventual consistency is used as a label without defining the expected final state, deadline, or recovery path. Temporary inconsistency then becomes permanent drift. Treating Broker Delivery as Business Correctness A broker can redeliver a message, but it cannot know whether the corresponding business effect already occurred. Correctness must be enforced where business state changes. Hiding Workflow State in Application Code Retries, timeouts, and compensations are implemented as nested calls without a durable state model. After a crash, the system cannot determine where to resume. Assuming Compensation Always Works The design contains a backward arrow for every forward step, even when the business action is time-sensitive or irreversible. Architecture Is the Placement of Guarantees The central question in distributed design is not whether a system uses microservices, events, sagas, or an outbox. It is where the system places its guarantees. Use a local transaction for the smallest set of state that must change together. Use a durable handoff when committed state must be communicated. Use an explicit workflow when multiple services must converge over time. Add compensation only where the business can genuinely support it. Most importantly, let business invariants shape the boundaries. A clean service diagram is valuable, but it is not more important than preserving the rules that make the system correct. Microservices make boundaries visible. Good architecture makes those boundaries honest.

https://dzone.com/articles/apache-flink-1Most teams working with Apache Flink think about stream processing: filter events, aggregate values, enrich records, join streams. This covers the majority of real-time data problems. But there is a specific class of problems that stream processing handles poorly: detecting meaningful sequences of events across time. Did this specific chain of things happen, in this order, within this time window? Complex event processing (CEP) is the tool designed for exactly this, and it works fundamentally differently from stream processing. CEP is a built-in capability of Apache Flink, available as open source and deployable at scales the old proprietary platforms could never reach. Yet it remains underused and misunderstood. This article is a practical guide to what CEP is, when to use it, how Flink implements it, and where it fits in modern data architectures, including agentic AI, which is covered in depth in the companion article. What Is Complex Event Processing? The simplest way to understand CEP is through a contrast with regular stream processing. Stream processing handles events continuously, one at a time or in windows. It answers questions like: What is the average transaction value over the last five minutes? How many login attempts has this user made in the last hour? These are stateful, real-time computations, but they operate on individual events or aggregates. They do not look for sequences. CEP answers a different class of question: did event A happen, followed by event B within 30 seconds, but not preceded by event C in the last 10 minutes? It detects patterns across multiple events in a defined order within a defined time window. Think of it as a regular expression engine applied to event streams instead of text strings. You define the pattern once. Flink evaluates every incoming event against every active pattern continuously. The moment a complete match is found, a result is emitted. The architectural implication is significant. In a traditional database, queries run against stored data. In CEP, data runs against stored queries. Events that match no active pattern are discarded immediately. This makes CEP extremely efficient at high volumes because irrelevant data never accumulates. One distinction worth making explicit: CEP uses static, predefined patterns. You specify exactly what sequence you are looking for. This is the right approach when the business logic is known and deterministic. The alternative is dynamic anomaly detection, where stream processing with statistical or ML functions builds a baseline from historical data and flags deviations without needing explicit rules. Both approaches solve real problems, and both run on Flink. The choice depends on whether you know the pattern you are hunting for. More on this in the section on when not to use Flink CEP. The Missing Event Problem One of the most valuable and least understood CEP use cases deserves its own section: detecting events that do NOT happen. This surprises most practitioners because the instinct is to think of event processing as reacting to events that arrive. But some of the most important business signals are absences. A machine sensor that should emit a heartbeat every 60 seconds goes silent. A delivery confirmation that should follow a shipment event within four hours never arrives. A payment settlement that should complete a transaction chain is missing. In each case, the business needs to know immediately, not after a human notices the gap. The instinct when facing this problem is to reach for the CEP Pattern API or MATCH_RECOGNIZE. But for missing event detection, a LEFT JOIN in Flink SQL is often the more appropriate and more memory-efficient approach. The query asks: give me all cases where event A arrived but the expected event B never followed within the time window. This is cleaner and more predictable for state management than expressing absence as a negative pattern. Memory management matters more here than most teams expect. Flink maintains state for every active pattern it is evaluating. A pattern with a seven-day lookback window across high-cardinality keys requires far more memory than one with a 60-minute window. Always use the WITHIN clause in pattern definitions to bound the time window explicitly. Without it, state grows unbounded in long-running streaming jobs, which is one of the most common operational problems in production Flink CEP deployments. Vendor Landscape: From Legacy Proprietary Platforms to Apache Flink Before looking at the vendors, one distinction is worth clarifying: the CEP market and the stream processing market were historically separate categories with separate products. The diagram below illustrates why: stream processing outputs on every window; CEP outputs only when a complete pattern match is confirmed. Understanding this separation explains both the fragmentation of the old market and why Flink's ability to handle both in one platform represents such a significant shift. TIBCO: BusinessEvents (CEP) and StreamBase (Stream Processing) A dominant middleware vendor of that era, TIBCO had two separate products. BusinessEvents, launched in 2004, was a pure CEP engine: rule-based inference, stateful event correlation, strong particularly in telco and insurance. StreamBase, acquired in 2013, was a completely separate pure stream processing platform designed for high-throughput financial trading. Software AG: Apama (CEP and Stream Processing in One) Software AG Apama was the notable exception: a single platform combining CEP and event stream processing in one product. This was its competitive advantage against TIBCO, which needed two separate tools to cover the same ground. Apama was strong in capital markets and algorithmic trading and competed directly in the financial services segment. Software AG acquired it in 2011. IBM and Oracle: Niche Adoption IBM WebSphere Business Events and Oracle CEP rounded out the market, though neither achieved the adoption depth that TIBCO or Apama built in their core verticals. All of the above platforms are effectively legacy today. They were expensive, required deep specialist expertise, and could not scale to the event volumes that modern architectures demand. Apache Flink arrived as an open-source framework that absorbed the core capabilities of all of them into a single unified platform running at far greater scale and at a fraction of the cost. Open Source: Apache Flink The foundation of Apache Flink remains open source. Self-managed, fully featured, and the basis on which all commercial distributions are built. The right choice for teams with strong engineering capabilities who want maximum control and flexibility. IBM: Confluent Cloud and Confluent Platform Confluent delivers Flink as a fully managed serverless service on Confluent Cloud as well as a self-managed distribution via Confluent Platform, both with Apache Kafka tightly integrated. CEP via MATCH_RECOGNIZE in Flink SQL is a first-class capability across both. Confluent Platform additionally supports the DataStream API, giving teams on-premises or in private cloud environments access to the full programmatic CEP feature set. Confluent was recently acquired by IBM, a move that positions the combined portfolio as a core pillar of IBM's hybrid cloud strategy: enterprises can run Confluent Cloud as a fully managed service or deploy Confluent Platform on-premises and across private clouds, with IBM's enterprise reach and support behind both. Ververica: VERA Engine Ververica, founded by the original creators of Apache Flink, offers the VERA engine. VERA follows an open-core model: 100% compatible with open-source Apache Flink, with proprietary performance optimizations on top. Key capabilities include updating pattern matching rules in a running job without restart, the Gemini state storage engine, which replaces the standard RocksDB backend with a cloud-native alternative delivering significantly faster snapshots, compute-storage separation, and streaming join optimizations that can reach up to twice the performance of vanilla Apache Flink for demanding workloads. Ververica is available in three deployment models: self-managed on-premises via Ververica Platform, fully managed on Ververica's public cloud, and BYOC where Ververica manages the platform on the customer's own cloud infrastructure. The managed cloud offering provides pay-as-you-go pricing billed hourly per compute unit, or reserved capacity for predictable pricing. It is not fully serverless in the scale-to-zero sense, making it better suited for teams with stable, continuous workloads than for highly variable or intermittent processing patterns. Amazon: Managed Service for Apache Flink Amazon Managed Service for Apache Flink is AWS's fully managed Flink offering and a natural choice for organizations already running on AWS infrastructure. The service handles cluster provisioning, auto-scaling, checkpointing, and multi-AZ availability, and it supports Java, Python, and SQL APIs, including MATCH_RECOGNIZE for CEP use cases. One important caveat on the "fully managed" positioning: like Amazon MSK, the service is not truly serverless. Billing is based on allocated KPUs (compute units) rather than actual consumption, meaning you pay for provisioned capacity regardless of whether your jobs are actively processing data. This is the same marketing gap that caught many MSK users by surprise. Teams deeply committed to the AWS stack who want Flink without managing cluster infrastructure will find it a reasonable option. Those needing a consumption-based, fully managed platform with deeper CEP and SQL capabilities will find Confluent Cloud or Ververica the stronger choices. For a complete map of the data streaming vendor landscape including deployment models and adoption rankings, the Data Streaming Landscape 2026 is the reference. How Flink Implements CEP Flink provides two complementary approaches to CEP. The right choice depends on pattern complexity and the technical profile of the team. One important note on distribution coverage: the Pattern API via the DataStream API is available in self-managed Apache Flink, Confluent Platform, and Ververica, but not in Confluent Cloud, which exposes Flink exclusively through the SQL and Table API. For teams on Confluent Cloud, MATCH_RECOGNIZE is the CEP interface available. The diagram below shows the core concept at a glance before diving into the technical details of each approach. The Pattern API via the DataStream API The programmatic approach, available in Java and Python. You define a Pattern object specifying the event sequence to detect, chaining conditions with operators: .next() for strict contiguity where events must follow immediately, .followedBy() for non-strict contiguity where other events can appear in between, and .within() for time constraints on the entire pattern. The Pattern is applied to a DataStream to create a PatternStream, from which matches are extracted using a PatternProcessFunction. This gives the most complete access to CEP capabilities and the most flexibility for complex logic. The tradeoff is that DataStream jobs require stronger engineering skills and can become difficult to maintain as complexity grows. MATCH_RECOGNIZE via the SQL and Table API The SQL approach is based on the ISO standard from December 2016 and available in Flink since version 1.7. Under the hood, it uses the same FlinkCEP library as the programmatic API. PARTITION BY handles per-entity pattern matching, ORDER BY ensures correct event ordering, PATTERN defines the sequence using regular-expression-style syntax, and DEFINE maps events to pattern variables. Flink adds the non-standard WITHIN clause for time-bounded patterns, which is essential for production memory management. The SQL approach is accessible to a broader range of developers, data engineers, and analysts. Its limitations relative to the full Pattern API are in practice a useful constraint: they prevent patterns from becoming so complex they are impossible to debug or maintain. For the majority of business-level CEP requirements, MATCH_RECOGNIZE is sufficient. The UI/UX Gap: CEP Is Still Largely an Engineering Discipline This is an honest limitation that most Flink CEP articles skip, and it deserves a direct discussion. Pattern Authoring Is an Engineering Task Pattern authoring in Apache Flink is fundamentally an engineering task. Writing DataStream API code in Java or Python requires a developer. Writing MATCH_RECOGNIZE SQL lowers the bar significantly but still assumes familiarity with SQL syntax, event time semantics, and the specific constraints of streaming pattern matching. A fraud operations analyst, a supply chain manager, or a compliance officer cannot walk up to a Flink cluster and define their own detection rules without technical support. This was not always the accepted norm in the CEP world. TIBCO BusinessEvents shipped with a visual rule editor and decision table support that allowed business analysts to author and maintain CEP rules without engineering involvement. Apama had a graphical development studio. These visual interfaces were real strengths of the proprietary platforms, enabling the domain experts who best understood the business patterns to directly encode them without going through an engineering ticket queue. The open-source Flink ecosystem has no mature equivalent today. The tradeoff is real but not entirely negative: engineering-based CEP is more powerful, more flexible, more testable, and more maintainable at scale than visual rule editors, which tend to produce opaque logic that is hard to version-control or audit. But it does gate adoption behind developer availability and creates a bottleneck when business rules need to change quickly. GenAI Is Closing the Gap, Partially The gap is starting to close, partially, through a different route than visual tooling. Current LLMs are capable of generating correct MATCH_RECOGNIZE SQL from a natural language description of a pattern. A fraud analyst describing "flag any user who makes more than three transactions over 500 euros within 10 minutes across different merchants" can get a working SQL pattern from a code-capable LLM today. This lowers the barrier for SQL-expressible patterns and gives non-engineers a practical path to prototyping detection logic before handing it to engineering for review and hardening. The limits are equally real. Generated DataStream API code for complex stateful patterns still requires engineering review before it goes anywhere near production. LLMs can produce syntactically correct Flink code that contains subtle logical errors in time semantics, state handling, or watermark behavior that are not obvious without deep Flink expertise. Generating and deploying without human review is not production-safe for mission-critical CEP logic today. The forward-looking picture is that GenAI will reduce but not eliminate the engineering dependency. Business analysts and domain experts working with well-defined, SQL-expressible patterns will be increasingly served by natural language interfaces, whether embedded in platforms like Confluent Cloud, Ververica, or third-party tooling built on top of Flink. Complex pattern engineering involving custom DataStream operators, iterative conditions, or deeply stateful logic will remain a developer discipline for the foreseeable future. The right platform architecture accounts for both populations: an abstraction layer exposing simple pattern authoring to business users, with full engineering-level access available for the cases that require it. Flink CEP Industry Use Cases CEP is not a niche capability. The following use cases represent industries where sequential pattern detection delivers business value that general stream processing cannot replicate. Financial Services: Fraud Detection Payment fraud is the most mature CEP application in production globally. Modern fraud is almost never a single suspicious event. It is a chain: a card used in Berlin, then two minutes later a contactless payment attempt in London, followed immediately by a high-value online transaction. CEP detects that sequence the moment it completes, calculates whether the travel time between locations is physically possible, and triggers a step-up authentication or account freeze before money moves. More sophisticated patterns chain five or more events: rapid card enumeration, velocity checks across channels, account takeover sequences. The ability to update detection rules in a running job without restart is particularly valuable in this domain. Fraud patterns evolve continuously with attacker behavior, and reducing the window between identifying a new attack pattern and deploying a rule to detect it is a direct business impact measure. Manufacturing and IoT: Predictive Maintenance A machine about to fail does not fail suddenly. It emits a multi-sensor signature in the period before failure: a subtle temperature rise, then increasing vibration frequency, then a pressure deviation. A single-sensor threshold alert would either miss the pattern entirely or produce too many false positives to be actionable. CEP detects the full multi-event signature reliably and triggers a maintenance workflow while there is still time to intervene, turning reactive maintenance into predictive maintenance. Event-driven architectures built on Apache Kafka and Apache Flink are already in production across automotive and industrial manufacturing environments for real-time operational intelligence. CEP-based predictive maintenance is a natural and high-value application layer on top of that existing streaming foundation, applying sequential pattern detection to the sensor streams that are already flowing. Supply Chain and ERP: Process Monitoring Business process events flowing from SAP and other ERP systems, B2B integration platforms, and EDI networks create a continuous stream representing the state of operations across the enterprise. A production order moving through its lifecycle generates events at every stage: order created, inventory checked, production started, quality inspected, goods dispatched. In B2B scenarios, the same logic applies across trading partner boundaries: a purchase order triggers an expected sequence of EDI messages including order acknowledgment, advance ship notice, and invoice. If any step is skipped, delayed beyond an SLA, or occurs out of order, the downstream business impact is real regardless of whether the event originates inside the enterprise or from an external partner. CEP monitors the entire process flow in real time across thousands of concurrent orders and trading partner transactions simultaneously. The moment a single order or EDI exchange deviates from its expected sequence, a structured alert is published to a downstream consumer and the appropriate workflow is triggered. This is categorically more valuable than a batch report that surfaces the deviation 24 hours after it occurred, when the window to intervene has already closed. Telecommunications: Network and Fraud Monitoring Telco was one of the original CEP industries and remains one of the strongest fits. Network intrusion detection, SLA breach monitoring across millions of simultaneous connections, and sequential call detail record analysis for subscription fraud all match the CEP pattern precisely. The combination of high event volumes, strict latency requirements, and the sequential nature of the fraud and failure patterns makes this a domain where CEP consistently delivers value that simpler stream processing cannot replicate. E-Commerce: Customer Journey Detection Sequential behavioral patterns are where MATCH_RECOGNIZE in Flink SQL particularly shines, because the patterns are expressible in clean SQL logic that sits close to the business domain. A customer who held a premium subscription, renewed twice, then downgraded to basic is a churn risk signal. A user who visited the same product page four times within a week without purchasing is a conversion opportunity. These patterns are straightforward to express in MATCH_RECOGNIZE and can be authored and maintained by analysts without requiring deep DataStream API expertise. The output feeds real-time personalization engines, retention workflow triggers, or conversion campaign systems. When NOT to Use Flink CEP CEP is a powerful tool for the right problems. For the wrong problems, it adds operational complexity with no benefit. Three situations where you should reach for something else: The pattern is undefined or too dynamic to specify explicitly. If you do not know what sequence you are looking for, or if the relevant patterns change faster than rules can be written and deployed, ML-based anomaly detection running on Flink's stream processing capabilities is the better approach. Let the model learn what normal behavior looks like from historical data and flag statistical deviations, rather than attempting to enumerate every possible bad pattern upfront. CEP and ML anomaly detection are complementary: CEP for known patterns, ML for unknown ones, both running on the same Flink platform feeding the same downstream systems. The problem does not actually require sequential pattern detection. If the pattern is expressible as a windowed aggregation rather than a strict event sequence, a standard Flink windowed stream processing query is simpler, cheaper, and easier to maintain than a CEP job. More than five failed logins within a 10-minute window, for example, is a count within a time window. It does not require CEP unless the ordering and interleaving of specific event types matters. CEP adds value specifically when sequence and ordering are part of what makes the pattern significant. If they are not, a tumbling or sliding window in Flink SQL handles it cleanly without the overhead of CEP state machines. And if the business action can wait for a scheduled report or nightly batch job, do not build a streaming architecture at all. The lookback window is impractically long relative to event volume. A pattern that requires tracking state across seven days of high-cardinality event data consumes significant Flink cluster memory. If the time window is very long and both event volume and key cardinality are high, the state management cost may outweigh the benefit. Either constrain the window more aggressively using the WITHIN clause, reduce key cardinality through pre-aggregation upstream, or reconsider whether a batch or lambda architecture is more appropriate for that specific pattern. Flink CEP Reference Architecture and Best Practices The reference architecture below shows the full picture: source systems feeding an event ingestion layer, Flink CEP as the pattern detector, and downstream consumers reacting independently to the structured output. The best practices that follow are derived directly from this architecture and from production deployments across industries. Keep patterns focused and always bound them with WITHIN. Every active pattern consumes Flink state proportional to its complexity and time window. Start with the simplest pattern that captures the business intent. Add the WITHIN clause in every pattern definition without exception. State that cannot be pruned will cause operational problems in production, often in ways that are difficult to diagnose under load. Decouple detection from response via a downstream consumer. The output of a CEP job should be a structured event published to a Kafka topic, a message broker, or whatever downstream system the architecture uses, not a direct action or API call embedded in the job itself. Downstream consumers react independently: alerting systems, workflow engines, AI agents. This makes the CEP job simpler and more stable, allows multiple consumers to react to the same signal independently, and lets response logic evolve without touching detection logic. It is the most important architectural principle for production CEP deployments. Avoid monolithic Flink jobs. A single large job combining pattern detection, enrichment, response logic, and downstream routing is an anti-pattern. A fraud detection job that also calls an external risk scoring API, writes to a database, and sends notifications is doing too many things. When any one component needs to change or fails, the entire job is affected. Separate concerns into focused jobs: CEP detection in one job, enrichment in another, AI inference in a third. Each job does one thing well and can be updated, scaled, and debugged independently. CEP Is Underused, Practical, and More Relevant Than Ever The decision framework is simple. CEP is the right tool when you know the pattern and need to detect it in real time. Windowed stream processing is simpler when ordering does not matter. ML-based anomaly detection is the right tool when the pattern is too dynamic to define explicitly. All three approaches are complementary and run on the same Flink platform. The legacy CEP market is gone. Apache Flink, delivered through Confluent (IBM), Ververica, and Amazon, has absorbed everything those platforms offered at scales and economics they could never reach. The remaining gap is the UI/UX barrier, which GenAI is beginning to close for SQL-expressible patterns but has not eliminated for complex logic. Detecting the pattern is only half the problem. The other half is what happens next: which system acts, with what context, and with how much autonomy. A follow-up article covers exactly that: how Flink CEP connects to Agentic AI architectures and why CEP is the layer that makes AI agents trustworthy rather than merely powerful. Stay tuned.

By Kai Wähner CORE

The project AngularAndSpring has been refactored from Spring Reactor to Spring blocking APIs using JDK 25 and virtual threads. The Old Architecture The reason AngularAndSpring was built using Spring Reactor was primarily the efficient use of operating system threads and memory under high load. Spring Reactor had a limited thread pool (usually the number of available CPU cores) and used OS threads only when the CPU had work to do. It did not let the operating system threads sleep. This used CPU resources much better and required far fewer OS threads, each of which needs around a megabyte of memory. The trade-off was a reactive programming style, which can become harder to read as the complexity of the implemented functionality grows, along with stack traces that are very difficult to read. These hard-to-read stack traces make finding and fixing errors much harder and significantly more time-consuming. How Virtual Threads Change the Trade-Off Before JDK 25, the choice was between efficient hardware use with Spring Reactor and easy-to-read code with easier debugging and readable stack traces. Spring's support for Java virtual threads offers the use of very efficient virtual threads for request processing. With virtual threads, operating system threads are used only when processing is needed; virtual threads are unmounted from the OS thread, freeing the OS thread to run a different virtual thread; and virtual threads have minimal memory requirements. That means a limited number of operating threads is able to handle a very large number of virtual threads. With JDK 25 and the virtual threads update, the choice has changed: JDK 25 with virtual threads uses hardware resources efficiently, and you can keep the code easy to read and debug with readable stack traces. There are limitations for virtual threads when native code is called, but the same limitations apply for Spring Reactor too. The main virtual threads blocker: The missing synchronized support was fixed in JDK 25. That means the main advantages of Spring Reactor are now provided by the JVM with its virtual threads. The New Architecture The basic building blocks of the new architecture can be found in the backend pom.xml. It now uses these starters: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-mongodb</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-restclient</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> The virtual threads are switched on with this in the application.properties: Properties files spring.threads.virtual.enabled=true The spring-boot-starter-webmvc is used to provide REST endpoints with blocking APIs. The spring-boot-starter-restclient is used to provide the new REST client for calling the blocking REST endpoints of other systems. The spring-boot-starter-data-mongodb is used to provide repositories and APIs for accessing MongoDB; all of these APIs are blocking. The ‘spring.threads.virtual.enabled=true’ enables the Spring Boot support for virtual threads provided by the JVM. The REST Endpoint The blocking REST endpoints now look like this in BitstampController: Java @RestController @RequestMapping("/bitstamp") public class BitstampController { private final BitstampService bitstampService; public BitstampController(BitstampService bitstampService) { this.bitstampService = bitstampService; } @GetMapping("/{currpair}/orderbook") public String getOrderbook(@PathVariable String currpair) { return this.bitstampService.getOrderbook(currpair); } ... } These endpoints do not return the reactor wrapper types. It just returns the intended return type. The ‘BitstampService’ methods now also return the intended return types without any wrappers. These endpoints just use plain Java. The Restclient The new blocking RestClient is now the Spring RestClient and looks like this in ScheduledTask: Java @Component public class ScheduledTask { private static final Logger LOG = LoggerFactory.getLogger(ScheduledTask.class); private static final String URLBS = "https://www.bitstamp.net/api"; private static final String URLCB = "https://api.coinbase.com/v2"; private static final String URLBF = "https://api.bitfinex.com"; private final BitstampService bitstampService; private final BitfinexService bitfinexService; private final CoinbaseService coinbaseService; private final MyUserService myUserService; private final RestClient.Builder restClientBuilder; public ScheduledTask(BitstampService bitstampService, MyUserService myUserService, BitfinexService bitfinexService, CoinbaseService coinbaseService, RestClient.Builder restClientBuilder) { this.bitstampService = bitstampService; this.bitfinexService = bitfinexService; this.coinbaseService = coinbaseService; this.myUserService = myUserService; this.restClientBuilder = restClientBuilder; } ... @Async("clientTaskExecutor") @Scheduled(fixedRate = 60000, initialDelay = 3000) @SchedulerLock(name = "BitstampQuoteBTC_scheduledTask", lockAtLeastFor = "PT50S", lockAtMostFor = "PT55S") public void insertBitstampQuoteBTC() { String currPair = "btceur"; insertBsQuote(currPair); } private ClientHttpRequestFactory customRequestFactory() { HttpClientSettings settings = HttpClientSettings.defaults() .withConnectTimeout(Duration.ofSeconds(4)) .withReadTimeout(Duration.ofSeconds(5)); return ClientHttpRequestFactoryBuilder.detect().build(settings); } private void insertBsQuote(String currPair) { LocalTime start = LocalTime.now(); try { QuoteBs quote = this.restClientBuilder.requestFactory(this.customRequestFactory()).build().get() .uri(String.format("%s/v2/ticker/%s/", ScheduledTask.URLBS, currPair)) .accept(MediaType.APPLICATION_JSON).retrieve().body(QuoteBs.class); quote.setPair(currPair); quote = this.limitPrecision(quote); this.bitstampService.insertQuote(quote); this.logDuration("Bitstamp", currPair, start); } catch (Exception e) { LOG.warn(String.format("Bitstamp data import failed for: %s", currPair), e); } } ... } The RestClient is created using the RestClientBuilder to allow connect and read timeouts to be configured. The timeouts are created in the customRequestFactory() method and set as a parameter of the RestClientBuilder. The RestClient uses the .uri(...) method to set the URL to call. The .accept(...) method sets the MediaType in the HTTP header that is expected from the REST client. The .retrieve() method executes the REST call, and the .body(...) method specifies the object into which the JSON response should be mapped. For debugging purposes, it can be useful to map the response body to a String first and analyze the response to ensure that it can be mapped correctly to the appropriate DTOs. The MongoDB Client For accessing the database, Spring Data MongoDB is used. The per-minute quotes are in the QuoteBs, QuoteBf, QuoteCb entities. The per-hour quotes are in the QuoteHourBs, QuoteHourBf, QuoteHourCb entities, and the per-day quotes are in the QuoteDayBs, QuoteDayBf, QuoteDayCb entities. The different entities enable Spring Data MongoDB support for each entity with its own repository. This is an example of QuoteBsMongoRepository: Java public interface QuoteBsMongoRepository extends MongoRepository<QuoteBs, ObjectId>, QuoteBsRepository { ... } The Interface ‘MongoRepository<QuoteBs, ObjectId>’ provides standard interfaces and enables the generated CRUD methods in QuoteBsRepository. Java public interface QuoteBsRepository extends QuoteRepository<QuoteBs>, QuoteMongoRepository<QuoteBs> { ... QuoteBs insert(QuoteBs quote); List<QuoteBs> findByPairAndCreatedAtAfterOrderByCreatedAtAsc(String pair, Date date, Limit limit); @Query("{ 'createdAt': { '$gt': ?0, '$lt': ?1 } }") List<QuoteBs> findByCreatedAtGreaterThanAndCreatedAtLessThan(Date from, Date to); } This interface uses the by-method-name-generated CRUD methods that Spring Data MongoDB provides and uses the ‘@Query’ annotation for a finder that can not be generated only by a method name. The entity QuoteBs looks like this: Java @Document public class QuoteBs implements Quote,QuotePairBs { @Id private ObjectId _id; @NotBlank @Indexed @JsonProperty private String pair; @NotNull @Indexed(name = "QuoteBs-createdAt") @JsonProperty private Date createdAt = new Date(); private BigDecimal high; private BigDecimal last; private Date timestamp; private BigDecimal bid; private BigDecimal vwap; private BigDecimal volume; private BigDecimal low; private BigDecimal ask; private BigDecimal open; The ‘@Document’ annotation marks the class as a MongoDB document. The ‘QuotePairBs’ interface is used to be able to access the common properties of minute, hour, and day entities of the same source. The ‘@Indexed’ annotation makes sure an index is created for this property in MongoDB for faster access and ordering. The precision of the Java BigDecimal type is higher than the precision of the MongoDB number type. Because of that, this method in ScheduledTask is used: Java private <T> T limitPrecision(T quote) { var fields = quote.getClass().getDeclaredFields(); for(Field field : fields) { if(field.getType().isAssignableFrom(BigDecimal.class)) { field.setAccessible(true); try { var value = (BigDecimal) field.get(quote); value = value.round(new MathContext(30, RoundingMode.HALF_UP)); field.set(quote, value); } catch (IllegalAccessException e) { throw new RuntimeException(e); } } } return quote; } The precision is limited to 30 decimals for all properties with the type BigDecimal. This is enough for all use cases in this application. Thread Pools If the hour and day quotes are recalculated from a large number of minute quotes, this can take several minutes. To avoid competing with the thread pool handling the user requests, a separate thread pool is created for this task in SchedulingConfig: Java @Bean(name = "clientTaskExecutor") public Executor threadPoolTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setVirtualThreads(true); executor.setThreadNamePrefix("clientTask-"); return executor; } This creates a separate TaskExecutor that can be used in the ‘@Async(“clientTaskExecutor”)’ annotations. That makes sure that no user requests wait in the queue for a free thread while long-running threads are at work. Conclusion With virtual threads in JVM 25 and Spring’s support for them, many reasons for using reactive programming in Spring have disappeared. The synchronized issue with virtual threads has been fixed in JDK 25, and the remaining causes of thread pinning are not relevant anymore for most use cases. The JVM now provides scalability and efficiency of reactive frameworks out of the box, while the blocking programming model is easier to implement and debug. Spring Data MongoDB support makes database access much easier with generated CRUD methods and the ‘@Query(…)’ support. It is time for projects still using reactive programming to move to an easier, faster programming model for developers.

The familiar claim that software behaves correctly on a developer workstation but fails elsewhere is often treated as a debugging cliché. In practice, it describes an architectural failure where part of the system’s behavior exists outside version control. Source code may be identical while the effective execution environment differs in runtime version, CPU architecture, filesystem semantics, locale, time zone, native libraries, certificates, environment variables, dependency resolution, or build tooling. When those inputs remain implicit, the developer workstation becomes an undocumented dependency. The resulting failure is not merely environmental inconvenience, but it is evidence that the system lacks a complete, reproducible definition of the conditions under which it is expected to build and run. Reproducible-build practices are based on the same principle: a declared source and build inputs should create an independently verifiable path to the resulting artifact rather than depending on hidden machine state. The Machine Is an Input, Even When the Repository Pretends Otherwise A repository normally captures application source, configuration templates, and dependency manifests, but operating assumptions often remain outside it. A service compiled with one JDK can behave differently under another. Code that relies on the host time zone inherits machine configuration, and Java’s default time zone is derived from the environment in which the JVM runs; APIs that obtain the system default can therefore return different values across hosts. Locale has a similar characteristic because a JVM maintains a default locale that can influence locale-sensitive operations. The safer approach is to make important defaults explicit at application boundaries. A timestamp formatted for interchange, for example, should not silently inherit workstation configuration: Java String timestamp = eventTime .atZone(ZoneOffset.UTC) .format(DateTimeFormatter.ISO_OFFSET_DATE_TIME); Using an explicit UTC offset removes the host time zone from this operation. The same principle applies to number parsing, character encoding, paths, sorting, and regional formatting. Defaults are convenient only while machines happen to share them. Architecture becomes more predictable when environmental choices are represented as declared inputs or explicit invariants. Java’s time API distinguishes instants, offsets, and full time-zone rules, including daylight-saving gaps and overlaps, making this distinction important whenever local time participates in application logic. Filesystem behavior creates a subtler form of skew. Git documents core.ignoreCasebecause filesystems including APFS, HFS+, FAT, and NTFS can require case-insensitive handling. Git probes this behavior when a repository is created or cloned and configures its operation accordingly. Consequently, a reference to Config.json may appear valid in one environment while deployment fails because the repository actually contains config.json. Reproducibility Starts Before Deployment Locking application dependencies is necessary, but it is not sufficient. The package manager, compiler, operating-system libraries, runtime, and build-time tools also participate in artifact production. npm’s npm ci, for example, requires an existing lockfile and exits when package.json and the lockfile disagree rather than modifying the lockfile to accommodate the current installation. That behavior makes dependency resolution more suitable for automated, repeatable builds. Shell node --version npm --version npm ci npm test These commands become significantly more meaningful when the Node.js and npm versions are controlled by CI rather than inherited from whatever happens to be installed. A successful workstation build should not define correctness as a clean build executed from declared inputs should. Hermetic build systems take this further. Bazel defines a hermetic action as one that consumes declared inputs rather than arbitrary files from the host filesystem and produces declared outputs. Sandboxing helps detect accidental dependencies on local tools or files. The architectural benefit extends beyond caching and build speed, as an undeclared dependency can become a build failure instead of remaining hidden until another machine lacks it. That changes an important relationship: Plain Text source + declared dependencies + declared toolchain ↓ artifact The workstation should not be an additional invisible operand in that equation. Containers Reduce Machine Drift, but They Do Not Erase It Containers are useful because they package a controlled userspace, but containerization does not make host differences disappear. CPU architecture remains significant. Docker treats linux/amd64 and linux/arm64 as separate platform targets and provides multi-platform builds for producing images across operating-system and architecture combinations. Native libraries, JNI dependencies, compiled Node modules, and architecture-specific executables can expose such differences immediately. A CI pipeline can turn architecture support into an executable requirement: Shell docker buildx build \ --platform linux/amd64,linux/arm64 \ --tag registry.example.com/payments:${GIT_SHA} \ --push . This is stronger than discovering an architecture mismatch after deployment because the supported platforms become part of the build contract. Base images introduce another source of environmental movement. Container tags are references that can point to newer image content over time. Docker recommends digest pinning when exact image identity matters, as referencing a digest ensures that the same image version is selected even if the associated tag later changes. Dockerfile ARG RUNTIME_IMAGE FROM ${RUNTIME_IMAGE} ENV TZ=UTC COPY service.jar /app/service.jar ENTRYPOINT ["java", "-Duser.timezone=UTC", "-jar", "/app/service.jar"] The pipeline can provide RUNTIME_IMAGE as an approved repository plus immutable digest. Base-image updates then become explicit changes that can pass through build, test, review, and deployment rather than appearing because a floating tag moved. Containers therefore narrow the environment boundary, and they do not eliminate the requirement to define it. Development Environments Belong in Source Control Development machines do not need to reproduce every operational detail of production, but the toolchain that determines software behavior should be describable. The Development Container Specification exists to capture development-specific tools, settings, runtime configuration, and metadata in a machine-readable definition such as devcontainer.json. JSON { "name": "payments-service", "image": "ghcr.io/example/java-dev@sha256:<approved-digest>", "containerEnv": { "TZ": "UTC", "LANG": "en_US.UTF-8" } } The important property is not this particular format. The architectural goal is that a clean machine can reconstruct the required toolchain without depending on manually installed compilers, remembered shell configuration, stale global packages, or undocumented IDE state. This distinction separates documentation from reproducibility. A setup guide describes actions that are expected to produce an environment. An executable environment definition participates directly in producing that environment. The latter is easier to validate continuously and considerably harder to misinterpret. Environmental Assumptions Should Be Observable Even controlled environments can drift. Production diagnosis becomes easier when runtime identity is recorded at startup alongside application version and build commit. Java exposes runtime, architecture, time zone, and locale information directly, allowing relevant execution characteristics to be captured without exposing arbitrary environment variables. Java log.info( "runtime java={} arch={} timezone={} locale={} commit={}", Runtime.version(), System.getProperty("os.arch"), ZoneId.systemDefault().getId(), Locale.getDefault().toLanguageTag(), buildCommit ); This fingerprint turns environment comparison into evidence. A defect occurring only on aarch64, under a particular time zone, or under a different runtime version can be correlated with actual execution metadata instead of machine descriptions passed through incident discussions. Critical assumptions can go further and become startup invariants: Java if (!ZoneId.systemDefault().equals(ZoneOffset.UTC)) { throw new IllegalStateException( "Unsupported runtime timezone: " + ZoneId.systemDefault() ); } Failing immediately is often safer than allowing an unsupported environment to produce subtly incorrect state. Testing should follow the same principle. Dimensions intentionally supported by the software should vary in CI, while dimensions that must remain constant should be frozen. Cross-platform builds validate CPU targets. Clean package installation validates dependency locks. Case-sensitive environments expose path errors. Explicit time-zone tests reveal accidental dependence on host defaults. The distinction is architectural, as supported variability is tested and unsupported variability is removed. Architecture Must Define the Machine Boundary The lasting solution to environment-specific failures is not a larger setup document or a longer troubleshooting checklist. It is a tighter definition of system inputs where locked dependencies, controlled toolchains, pinned base images, explicit locale and time-zone behavior, validated CPU architectures, hermetic builds where practical, reproducible development environments, and observable runtime identity. Reproducible Builds describes reproducibility as an independently verifiable path from source to binary, and the same engineering discipline can be extended to the environment in which that binary executes. A reliable software architecture therefore includes more than source code, services, databases, and deployment topology. It also includes the assumptions connecting code to a machine. When those assumptions are declared, versioned, tested, and observable, environment-specific failures become ordinary engineering defects with traceable causes. When they remain implicit, every workstation quietly becomes a different architecture.

By Uthej Mopathi CORE

This is Part 3 of a series on running sentiment analysis inside Neo4j. Part 1, Running Sentiment Analysis Inside Neo4j With a Java Plugin, used the VADER lexicon via a pure Java UDF – rule-based, fully local, high throughput.Part 2, Wasm Inside Neo4j: Building the Example That Didn't Exist, embedded a Wasm runtime inside a Java UDF using wasmtime-java — sandboxed, still local, lower throughput. Part 3 takes a different approach. Instead of bundling a scoring model inside the plugin JAR, we call TypeSafe AI's Jev from inside a Java UDF. Jev is a System One model – it doesn't generate text. It takes a state (the text to evaluate) and a set of typed questions and returns structured decisions with explicit probability and confidence information. No prompt engineering, no JSON schema coercion, no parsing. The result is callable directly from Cypher: Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; The full source code is available on GitHub. What Jev Gives You That VADER Doesn't VADER gives you a compound score. For "The movie had some good moments but ultimately disappointed," it returns something around -0.3. That's a number, but it doesn't tell you how much to trust it. Jev separates the decision from the uncertainty. For the same sentence, it returns: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score — which is the result of the separate confidence question we asked Jev, rated on a scale from 0 to 2 — is 1.55, landing between "Somewhat confident" and "Very confident." The separate confidence score provides an additional signal about the classification and that information can be actionable in a way that a raw sentiment score isn't. For unambiguous text, Jev is equally unambiguous: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, ... } This structured uncertainty output is the key differentiator of Part 3. If you need to act differently on low-confidence classifications — routing them to human review, flagging them for follow-up, or simply surfacing the uncertainty in a UI — VADER's scores don't provide a model-derived estimate of uncertainty about the classification. Jev returns an explicit confidence assessment alongside the decision. Confidence information should not be confused with accuracy: a model can be highly confident and still be wrong. What We're Building A Neo4j Java UDF that calls the Jev API over HTTPS, passing the input text as the evaluation state and asking two typed questions: a sentiment classification (positive, negative, or neutral) and a confidence score. For non-blank input, the UDF returns a Map<String, Object> with six keys: choice, confidence, score, positive, negative and neutral. For blank input, it returns only choice and confidence as a guard response without calling the API. No Wasm runtime, no bundled lexicon, no native libraries. The plugin has no Wasm runtime, native library, or bundled sentiment lexicon. Its only runtime dependency beyond Neo4j itself is Jackson, which is relocated into a private package namespace to avoid classpath conflicts with Neo4j's own Jackson dependencies. Prerequisites Java 21. OpenJDK 21 or later. Confirm with java -version.Maven 3.9.6. Install manually rather than via Homebrew on Apple Silicon. Confirm with mvn -version.Neo4j Desktop. Download and install Neo4j for Desktop. We're using version 2026.07.0. Update the neo4j.version property in pom.xml to match your installation.A Jev API key. Sign up for access at TypeSafe AI. The key is passed to Neo4j via a JVM property in neo4j.conf — more on this in the deployment section. Getting the Code The full source is on GitHub. Clone the repo and navigate to the neo4j-jev-udf directory: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set jev-udf mv jev-udf ../jev-udf cd ../jev-udf/neo4j-jev-udf The Maven Project The pom.xml has two things worth noting. First, org.neo4j:neo4j is declared as provided scope — Neo4j is already present in the database JVM at runtime. Second, we use maven-shade-plugin to relocate Jackson into a private package namespace. Neo4j ships its own version of Jackson internally, and without relocation, the two can conflict at runtime in ways that produce silent failures. XML <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>jev-neo4j-poc</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>2026.07.0</neo4j.version> </properties> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.17.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <relocations> <relocation> <pattern>com.fasterxml.jackson</pattern> <shadedPattern>com.example.jev.shaded.jackson</shadedPattern> </relocation> </relocations> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Update the neo4j.version property to match your own Neo4j Desktop installation. The Java UDF The UDF uses Java's built-in HttpClient — no third-party HTTP library needed. The Jev API key is read first from the environment variable TYPESAFE_API_KEY, then from a JVM system property of the same name as a fallback. The fallback matters because Neo4j Desktop runs as a separate process and doesn't inherit your shell's environment variables; we'll set the JVM property in neo4j.conf in the deployment section. Java package com.example; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ArrayNode; import com.fasterxml.jackson.databind.node.ObjectNode; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.util.HashMap; import java.util.Map; public class JevUDF { private static final String API_URL = "https://api.typesafe.ai/v1/systemone"; private static final ObjectMapper mapper = new ObjectMapper(); private static final HttpClient client = HttpClient.newHttpClient(); private static String getApiKey() { String key = System.getenv("TYPESAFE_API_KEY"); if (key == null || key.isBlank()) { key = System.getProperty("TYPESAFE_API_KEY"); } if (key == null || key.isBlank()) { throw new RuntimeException("TYPESAFE_API_KEY not set as environment variable or JVM property"); } return key; } @UserFunction("com.example.jev.sentiment") @Description("Classifies text sentiment using Jev (TypeSafe AI). Returns choice, probabilities and confidence.") public Map<String, Object> sentiment(@Name("text") String text) throws Exception { if (text == null || text.isBlank()) { Map<String, Object> empty = new HashMap<>(); empty.put("choice", "neutral"); empty.put("confidence", 0.0); return empty; } String apiKey = getApiKey(); ObjectNode body = mapper.createObjectNode(); body.put("model", "jev-latest"); body.put("state", text); ObjectNode questions = mapper.createObjectNode(); ObjectNode sentiment = mapper.createObjectNode(); sentiment.put("type", "choice"); sentiment.put("instructions", "What is the overall sentiment of this text?"); ObjectNode criteria = mapper.createObjectNode(); criteria.put("positive", "Text expresses positive sentiment"); criteria.put("negative", "Text expresses negative sentiment"); criteria.put("neutral", "Text is neutral or factual"); sentiment.set("criteria", criteria); questions.set("sentiment", sentiment); ObjectNode confidence = mapper.createObjectNode(); confidence.put("type", "score"); confidence.put("instructions", "How confident are you in the sentiment classification?"); ArrayNode levels = mapper.createArrayNode(); levels.add("Not confident"); levels.add("Somewhat confident"); levels.add("Very confident"); confidence.set("criteria", levels); questions.set("confidence", confidence); body.set("questions", questions); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Content-Type", "application/json") .header("Authorization", "Bearer " + apiKey) .POST(HttpRequest.BodyPublishers.ofString(mapper.writeValueAsString(body))) .build(); HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) { throw new RuntimeException("Jev API error " + response.statusCode() + ": " + response.body()); } JsonNode result = mapper.readTree(response.body()); JsonNode answers = result.get("answers"); JsonNode sentimentAnswer = answers.get("sentiment"); JsonNode confidenceAnswer = answers.get("confidence"); Map<String, Object> output = new HashMap<>(); output.put("choice", sentimentAnswer.get("choice").asText()); output.put("confidence", sentimentAnswer.get("confidence").asDouble()); output.put("score", confidenceAnswer.get("score").asDouble()); output.put("positive", sentimentAnswer.get("probabilities").get("positive").asDouble()); output.put("negative", sentimentAnswer.get("probabilities").get("negative").asDouble()); output.put("neutral", sentimentAnswer.get("probabilities").get("neutral").asDouble()); return output; } } A few things worth noting in the Jev request body. The choice question uses a criteria object — a map of option names to descriptions. The score question uses a criteria array — an ordered list of level descriptions. These are different shapes, and mixing them up produces a 422 error from the API. Both questions are answered in a single API call; Jev evaluates them in parallel against the same state. Deploying to Neo4j Build the JAR: Shell mvn -q clean package Stop Neo4j in Desktop, then copy the JAR to your plugins folder: Shell cp target/jev-neo4j-poc-1.0-SNAPSHOT.jar \ ~/Library/Application\ Support/neo4j-desktop/Application/Data/dbmss/<your-dbms-id>/plugins/ On Linux, the plugins folder is typically under $NEO4J_HOME/plugins/. On Windows, the plugins folder is typically under %NEO4J_HOME%\plugins\. Edit neo4j.conf in the conf/ folder of the same dbms directory and add three lines: Plain Text dbms.security.procedures.unrestricted=com.example.jev.* dbms.security.procedures.allowlist=com.example.jev.* server.jvm.additional=-DTYPESAFE_API_KEY=<your-api-key> Two lines are shown in the security configuration. allowlist controls which extension functions are loaded and makes the function callable from Cypher. unrestricted gives matching extensions full database access, bypassing Neo4j's security restrictions and should be used only where necessary. For this example, both settings are configured for the com.example.jev.* namespace; use the narrowest configuration that your Neo4j version and deployment require. The server.jvm.additional line passes the API key to Neo4j's JVM as a system property. For this local Neo4j Desktop example, that's a practical way to make the key available without modifying the UDF code. The Neo4j Desktop process does not automatically receive environment variables exported in your terminal session, so the JVM property is the reliable alternative. Do not commit this value to source control. For production, use your deployment environment's secret-management mechanism and inject the value into the Neo4j process without storing the secret in a tracked configuration file. Restart Neo4j in Desktop, then select Query from the left-hand navigation pane, connect to your instance, and confirm the function is registered: Cypher SHOW FUNCTIONS YIELD name WHERE name STARTS WITH 'com.example' RETURN name; Expected: "com.example.jev.sentiment" Results Positive Sentence Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; Expected: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, "positive": 1.0, "negative": 0.0, "neutral": 0.0 } Negative Sentence Cypher RETURN com.example.jev.sentiment('The movie was absolutely terrible.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 2.0, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } Mixed Sentiment Cypher RETURN com.example.jev.sentiment('The movie had some good moments but ultimately disappointed.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score drops to 1.55 out of 2 – somewhere between "Somewhat confident" and "Very confident." This is structured uncertainty output, and it's what distinguishes Jev from a rule-based scorer. There are two different quantities here. confidence belongs to the sentiment choice itself: it describes the probability concentration of the classification. score comes from a separate question that explicitly asks Jev how confident it is in that classification. They answer different questions and should not be interpreted as two versions of the same confidence value. Blank-Input Guard Cypher RETURN com.example.jev.sentiment('') AS result; Expected: JSON { "choice": "neutral", "confidence": 0.0 } The blank-input guard returns immediately without making an API call. Here neutral is a guard value rather than a Jev classification: no sentiment analysis is performed for blank or whitespace-only input. Scalability and Limitations Every call to com.example.jev.sentiment() makes an outbound HTTPS request to the Jev API. That shapes where this approach fits and where it doesn't. Latency. Jev responds in 70-500ms per call. For an occasional interactive request — a customer submitting a review, a user looking up a specific node — that latency may be acceptable. For a batch pipeline, if 1,000 calls are made sequentially at a 200ms average, that's over three minutes of API time; concurrent execution changes the wall-clock time but introduces its own rate-limit and resource considerations. Cost. At the pricing available when this article was written, input tokens cost approximately $0.042 per million and output tokens were free. The test requests in this article averaged approximately 381 input tokens per call, giving a per-call cost of around $0.000016 (381 ÷ 1,000,000 × $0.042). Pricing is subject to change; check TypeSafe AI's current pricing before estimating production cost. At this workload size, cost is unlikely to be the primary constraint; latency, throughput, network dependency, and data residency are more significant considerations. At that rate, 1,000 calls would cost roughly $0.016, while one million calls would cost roughly $16, assuming the same average input size. Network dependency. Parts 1 and 2 of this article series are fully local – no network, no external dependency, no data leaving the database. Part 3 introduces all three. The UDF itself runs inside Neo4j, but the inference does not: the text crosses the network boundary to TypeSafe AI, and only the structured result returns to Neo4j. If the Jev API is unreachable, the UDF throws. In a production deployment, that's an operational risk worth planning for: the HttpClient in this example has no explicit timeout configured; in production, set a connect timeout and a per-request timeout to prevent a slow or unresponsive API from blocking Neo4j worker threads indefinitely. A fallback return value and monitoring on the call path are also worth considering. Note also that the UDF makes a synchronous blocking HTTP call inside database query execution; concurrent Cypher invocations increase pressure on Neo4j worker threads, the API's rate limits, and the network. Data residency. Text passed to com.example.jev.sentiment() leaves Neo4j and is sent to TypeSafe AI's API. If your data residency requirements prohibit that, this approach isn't suitable. Review TypeSafe AI's data handling policies before deploying in a regulated environment. Caching as a mitigation. For workloads where the same text is scored repeatedly, results can be stored as node properties on first call and read directly on subsequent queries — skipping the API entirely. That's a natural pattern in Neo4j and largely eliminates the latency and cost concerns for stable datasets. The right use case. This approach is best suited to low-frequency, interactive workloads where explicit uncertainty information matters more than throughput and where data leaving the database is acceptable. For high-volume batch ingestion, Part 1 (the Java VADER UDF) remains a strong fit — local, fast, no network dependency. Part 3 trades local throughput and offline operation for a richer decision output and explicit uncertainty information. Comparing the Three Approaches Part 1: Java VADERPart 2: Wasm/VADERPart 3: JevOutputPolarity scores (float)Polarity scores (float)Typed decision + explicit uncertainty informationLocationFully localFully localNetwork (TypeSafe AI API)Latency< 1ms50-200ms (init)70-500msCostFreeFree~$0.000016/callOfflineYesYesNoExecution boundaryJava UDFWasm runtimeExternal APIUncertainty outputNoNoYesTypical fitHigh-volume batchSandboxed local inferenceInteractive, uncertainty-aware decisions Summary We added a third option to the series: calling TypeSafe AI's Jev from inside a Neo4j Java UDF, returning a typed sentiment decision with explicit uncertainty information directly from Cypher. The implementation has the smallest inference footprint of the three – no Wasm runtime, no bundled lexicon, no native libraries. The JAR contains only the UDF class, a relocated Jackson, and Java's built-in HttpClient. The main configuration consideration is passing the API key to Neo4j's JVM via server.jvm.additional in neo4j.conf, since the Neo4j Desktop process does not automatically receive environment variables exported in your terminal session. The key contribution of Part 3 isn't the mechanics — it's the explicit uncertainty output. For clear-cut sentences, Jev is unambiguous. For genuinely mixed sentiment, it returns both the decision and a structured confidence measure that Cypher can act on directly. That's something neither VADER nor the Wasm implementation provides. A deeper discussion of putting Jev inside Neo4j can be found in the free online book Lessons Learned Putting Jev Inside Neo4j. The full source code is available on GitHub.

By Akmal Chaudhri CORE