Upgrading to BeanShell version 3
Patchworks is changing to upgrade the BeanShell scripting language from version 2 to version 3. For the most part your BeanShell scripts will work exactly the same, but there are a few incompatibilities. Read this article to find out what you need to know to prepare for this switch.
Patchworks is a "toolkit" type of model having a large set of modeling components. These components can be assembled in to a custom configuration best suiting the modeling problem to be addressed. The software architecture consists of the core Patchworks program that handles the tasks of model initialization, scheduling of treatments, and synchronization of the various program elements and viewers. A large part of the toolkit is embodied in the Applications Programming Interface (API), a library of classes and methods used to create modeling structures including accounts, reports, map layers, scenario descriptions, and other tools. BeanShell is the extension language that is used to glue everything together. Patchworks is able execute BeanShell scripts that instantiate API classes and call API methods that modify structures or adjust modeling parameters.
One of the essential BeanShell scripts is the PIN (Patchworks Initialization) file. The PIN file sets global variables describing modeling input file locations, adjusts the accounting structure that tracks stand attributes and goals, and sets the initial user interface, including the initial set of map layers and reports. In a well organized, modular-style project the PIN file may invoke other scripts that perform related functions (such as initializing road modeling reports). Another important set of BeanShell scripts controls the setup and execution of scenarios. Although it is easy to carry out ad-hoc scenarios, scripts provide an explicit record of objectives and the foundation for embedded metadata describing the scenario purpose as well as details about parameter settings. It is a Patchworks best practise to maintain a modular and well documented set of TargetDescriptions and ScenarioDescriptions used to run scenarios on the study area.
Up until recently Patchworks has been using a slightly modified version of the BeanShell version 2.0b5 software. This version of BeanShell was released over 20 years ago. While it is robust software, it has not kept up with the changing times, the biggest limitation being that it cannot run on any version of Java newer than Java 8, which was first released in 2014. The underlying java platform has undergone many changes and improvements since then (for example, better handling of HiDPI monitors) , but Patchworks has been held back from taking advantage of these advanced features because of the BeanShell v2 compatibility problem.
Over the past few years work has been progressing on version 3 of the BeanShell software by a small team of open source volunteers. This new BeanShell version fixes many bugs and adds many modern features. Most importantly, it is compatible with the very latest Java platform improvements. Because of these factors it is important for Patchworks to pay it's technical debt and make the upgrade to the new version. Starting now, new versions of Patchworks will include BeanShell version 3.
Spatial Planning System will continue to support Patchworks using Beanshell v2 for the next year as customers make the switch, but the BeanShell v2 version will only include bug fixes and no new features. Customers will be able to switch between versions by reinstalling from the appropriate setup program as required. If you are experiencing a pain-point in the conversion to BeanShell v3 then it should be easy to downgrade to v2 and carry on. In addition, support will be available to all customers for the conversion of scripts from v2 to v3. If you are having a problem then get in touch with support@spatial.ca and we will provide priority assistance.
In my experiments over the past few months I have found that the vast majority of BeanShell scripts will run exactly as before, and for the most part the changes and new features in BeanShell are positive improvements. However, both BeanShell and the new Java platforms bring a few minor and unavoidable incompatibilities that I am aware of that may show up with existing scripts. I don't expect that these known issues will be much of an issue to deal with because they are fast fail: they show up immediately, they are very easy to fix, and the fixes required are backwards compatible to the BeanShell v2 releases.
Known issues upgrading from BeanShell v2 to BeanShell v3
In the rest of this article I will list the known incompatibilities between BeanShell v2 and v3, as well as what is required to adjust existing scripts to work with v3. If you are experiencing problems then check this list for a fix. I will keep this list updated in case new issues emerge.
Lexical scoping
The biggest backwards compatibility issue is that lexical scoping has been tightened. Lexical scoping describes how variables that are declared in one part of a program may or may not be visible in a different part of a program. In particular, lexical scopes are defined by program blocks, which in Java and BeanShell are often delimited by braces. When the BeanShell interpreter attempts to look up the value of a variable it will first search in the current lexical scope, and if not found then it will search upwards through the parent lexical scopes until it reaches the top global scope. The value of the variable in the closest upwards scope will be returned.
In the following example a variable 'a' is declared in the top level (global) scope. The 'if' statement introduces a program block and provides a new lexical scope. In this example the program block is delimited by braces, but if statements may also consist of a single statement not enclosed by braces. The inner lexical scope declares a new variable 'b'. In Java and many other languages, the variable 'b' will exist in the inner lexical scope and only be visible within that program block (and any other enclosed blocks). However, BeanShell v2 uses an irregular rule that variables declared in blocks will be installed in the parent lexical scope, so that these variables will be visible to that outer program scope.
a = 1;
if (a == 1) {
b = 2;
}
print(b); // this will work in BeanShell v2 but not v3
This is unlike most other languages, and frankly it's a bit weird. At the time that BeanShell v2 was written this was thought to be a convenience to simplify scripted code by reducing the number of variable declarations that are required, but now the thinking is that this is a bad idea. In BeanShell v3 'b' is not defined in the outer lexical scope and the print statement will throw an error about an undefined variable. To get the equivalent behaviour with Bsh3 you have to define b in the outer lexical scope:
a = 1;
b = 0;
if (a == 1) {
b = 2;
}
print(b); // this will work in both BeanShell v2 and BeanShell v3
There may be some places in your old BeanShell code where these issues will crop up, particularly in PIN files where 'if' statements are conditionally used to set program variables. These locations should be easy to find when using BeanShell v3 because there will be explicit error messages that the variable is not found. Here is an example from a PIN file:
if (zone.equals("X123")) {
caribouVersionChoice = select("Choose the HTS option:", "Choose caribou version",
new Integer[] {1,2,3,4,5}, 0);
caribouVersion = (caribouVersionChoice==null)?1: caribouVersionChoice;
} else {
caribouVersion = 1;
}
The above script works in BeanShell v2 because the variable 'caribouVersion' will be declared in the outer lexical scope. It will fail in BeanShell v3 because the 'caribouVersion' variable will only be visible within the lexical scope where it is declared. An easy fix for this situation is to rearrange the code to declare the variable in the outer scope:
caribouVersion = 1;
if (zone.equals("X123")) {
caribouVersionChoice = select("Choose the HTS option:", "Choose caribou version",
new Integer[] {1,2,3,4,5}, 0);
caribouVersion = (caribouVersionChoice==null)?1: caribouVersionChoice;
}
The above code will work in both BeanShell v2 and v3. The first statement defines the variable in the outer scope, and within the 'if' statement the variable lookup correctly find 'caribouVersion' in the outer scope.
In some special cases it may be undesirable to unconditionally declare a variable in an outer scope. For example, it is possible to set up batch scripts to run a PIN file. In this case the PIN file can check if a variable has already been defined by the parent script, and if not then interactively query for the value. Explicitly defining the variable at the top level of the PIN file would clobber the value from the batch file, so the definition must only happen if it does not already exist. Have a look at the next example:
if (zd == void || zd == null) {
print("Select the area to load");
zd = select("Choose the FMU or zone to load:","Choose FMU or zone", zones, 0);
if (zd == null)
return;
}
In this case the 'if' statement checks if the 'zd' variable does not exist or if it exists but does not have a value. If not existing, the script will interactively prompt for a value. In this case the 'zd' variable is being set inside the lexical scope, so in BeanShell v3 it will not be visible to the outer scope. We can explicitly set the variable in the outer lexical scope by using the 'super' qualifier. The 'super' qualifier accesses the parent lexical scope:
if (zd == void || zd == null) {
print("Select the area to load");
super.zd = select("Choose the FMU or zone to load:","Choose FMU or zone", zones, 0);
if (zd == null)
return;
}
The above code will work in both BeanShell v2 and v3. If 'zd' does not exist then 'super.zd' defines the variable in the outer scope, as we want.
Note that 'super' qualifiers may be chained together (super.super will access the parent of the parent). There is also the 'global' qualifier that will access the top level lexical scope, but be careful to not add too much variable pollution to the global scope.
Maintaining the old lexical scoping behaviour
Many scripts will have no issues with the new lexical scoping behaviour because they do not have conflicts with the new rules. Some scripts may have a few conflicts that are easy to identify and fix. However, some large, complex scripts may have many issues scattered over many files, and corrections may be tedious. For this last set of cases the Patchworks version of BeanShell v3 has a method to restore the old behaviour and maintain compatibility with v2 lexical scoping.
Add these lines to the top of the PIN file to enable compatibility:
if (this.interpreter.VERSION.startsWith("3")) {
this.interpreter.setBsh2ScopingCompatibility(true);
this.interpreter.getParent().setBsh2ScopingCompatibility(true);
}
The compatibility hack applies immediately as the method is invoked, and will apply to all scripts that are subsequently executed. There is no need to invoke this script more than once per program execution. It is also possible to turn the lexical scoping hack off again, perhaps only enabling it during a short section of problematic code.
The tighter v3 lexical scoping rules are a programming improvement, so this hack should be used judiciously. First try to adapt to the new rules. Only use this compatibility hack for legacy project where it would be inconvenient to make all of the required changes.
Undefined variables
Another issue is the handling of undeclared variables. In BeanShell v2 the use of undeclared variables would silently be accepted. In BeanShell v3 if you try to use an undefined variable you will get an error message. For example, lets assume that a variable named 'xxx' has not been declared. In BeanShell v2 the following command:
print(xxx);
will print
void
But if you try the same command in BeanShell v3 the script will be interrupted and you will get an error message containing:
throws Interpreter error undefined variable 'xxx'
This new behaviour is an improvement because typos like this that are silently ignored can cause hard to find bugs.
Default working directory
As the Java language has evolved the level of security has been tightened. In newer versions of the Java platform some of the clever tricks and sneaky shortcuts have been disabled. One of these that was being used by Patchworks was to alter the default working directory to match the location of the PIN file. This was done to simplify the calculation of relative file paths. A relative file path describes how to find a file relative to a default location. Relative file paths have names that do not start with a file path separator character (usually a /). Relative file paths can include folder names and may also ascend in to a parent folder by using the '..' notation. Here are some examples of relative file paths:
source("../scripts/reportConstants.bsh");
fmus = GeoRelationalStore.open("../data/fmus.shp");
These formats still work exactly the same under BeanShell v2 or BeanShell v3. There is a different usage that more than likely no longer works:
if (new File(tracks_path_prefix+"strata.csv").getAbsoluteFile().exists())
stratas = tracks_path_prefix+"strata.csv";
This uses the Java API to resolve the path relative to the default working directory. Patchworks can no longer switch the default working directory behinds the scenes to the PIN file folder, so the default working directory will be whatever was the default when the Patchworks program was launched. Often this folder will be the user "Documents" folder, and the relative path lookup will be incorrect from that location.
The work around for this problem is to use a Patchworks supported API to resolve relative paths using methods from the AttributeStore class:
if (AttributeStore.absoluteFile(tracks_path_prefix+"strata.csv").exists())
stratas = tracks_path_prefix+"strata.csv";
Choice of Java version
At the moment the BeanShell v3 version of Patchworks may be run on any Java platform from Java 8 to Java 22. During your transition to the BeanShell v3 version you may wish to stay with Java 8 in order to simplify the process of downgrading to BeanShell v2 (if required). However, Java 17 provides better support for HiDPI displays. If you are having display issues on high resolution monitors (menu fonts too tiny or too large) upgrading will provide a quick fix.
As new features are added to Patchworks the minimum Java version may change. There are features in the Java 22 that may enable a significant improvement in data interoperability, so stay tuned.