Replies: 5 comments 12 replies
-
|
Hello @jsorel, Yes, Apache SIS was listed in our proposal when joining the incubator of the Apache Foundation. I think SIS has the potential to become a very good alternative to GeoTools. Our main use case is the importation of vector data (shapefile, GeoPackage, etc.) into a database for producing vector tiles. I conducted a few experiments last year with Geotoolkit, but couldn't merge them due to license restrictions. As the shapefile reader in Apache SIS is unstable and has not been released, I ended up copying some of its sources into Apache Baremaps. I'd love to rely on Apache SIS and GeoAPI for this in the future. That being said, I found the feature API quite complex and believe the amount of work to implement a codecs is quite significant. A second use case I have in mind is the handling of a raster digital elevation models for producing contour lines and vector hillshades. From what I recently read, Apache SIS will soon provide a Cloud Optimized GeoTIFF reader, and the isoline tracer looks really interesting. I hope I will find time to work on this topic soon. On your side, have you identified some areas on which the two project overlap and should exchange or collaborate? Best, Bertil |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Thanks for the informations. Here are some news that might be of interest (I'm PMC on Apache SIS, working with Martin Desruisseaux)
About shapefile : A quick example : About exchange and collaboration :
I haven't look in very deep in Baremaps but you surely have symbology (OGC SE or else), some coverage support maybe ? The question is more : |
Beta Was this translation helpful? Give feedback.
-
|
The aim of Baremaps is to create open and user-friendly infrastructure components for web mapping (maps, search, geocoder, reverse-geocoder, IP to location, etc.). We do not intend to implement codecs for geodata. However, to achieve our goal, we often integrate or rewrite components when we cannot find off-the-shelf and ready-to-use solutions that meet our needs in terms of performance, usability, licensing, etc. Regarding the shapefile reader and writer, please do not hesitate to contact us if you wish to discuss this. Some of the simplifications we applied in Baremaps, such as the removal of the JDBC API, may be contributed upstream. I had not realized that Apache SIS could now replace Proj4j. We contributed some changes upstream to Proj4j to address EPSG licensing issues. Does Apache SIS address this problem? The classes we have for streaming PBF, OSM.XML, and OSC.XML files are robust and efficient. It may make sense to integrate them into Apache SIS in the future or to create an adapter for the FeatureSet API. However, I am unsure if OpenStreetMap changesets could be represented in Apache SIS. More generally, I think it would be beneficial to have low-level APIs for each targeted format (shapefile, geopackage, etc.) and then high-level adapters for the FeatureSet API. By doing so, integrators could use either the low-level API (for better performance and access to more functionalities) or the FeatureSet API (for interoperability and data transfer). For example, the geopackage reader of the National Geospatial-Intelligence Agency has a robust API and has an MIT license. It should be feasible to write an adapter for Apache SIS based on their API. To further clarify my previous answer regarding complexity, what we really miss in baremaps is simple and fast GeoDataFrame-like API to access the content of files as tabular data. |
Beta Was this translation helpful? Give feedback.
-
|
Hello all
Actually Apache SIS has referencing services since 2015. In EPSG terminology, it is a late-binding implementation. PROJ4 and PROJ4J by contrast are early-binding implementations, which was a cause of errors ranging from a few centimetres to 1 or 2 metres. It was a PROJ4 design flaw with profound implications, not a simple bug. PROJ 5 and latter resolved their issue with a 144,000$ fund raising by taking inspiration from Apache SIS (search "SIS" on GDAL barn page). But PROJ4J to my knowledge is still based on the PROJ4 design. Apache SIS provides EPSG data in a separated JAR files since the beginning. The separated JAR file uses the Alternatively, if doing a JavaFX or a console application, Apache SIS also provides mechanisms for interactively propose to the users to download that JAR file when first needed. The EPSG terms of uses are shown to the users, and if they click on "yes" on the question "Do you accept those terms", the JAR file is automatically added to the module path. That mechanism may need to be tweaked for each application however. |
Beta Was this translation helpful? Give feedback.
-
|
Hello again, The shapefile part in SIS has been revisited, with as you asked, both a high level API (WritableFeatureSet) and a low level one (Shp, Dbf, Shx, Cpg, readers and writers). Here is the javadoc with read/write code examples in each package. Could you have a look and tell me if it seems okay for your needs ? |
Beta Was this translation helpful? Give feedback.
-
|
(Not speaking for Bertil but just giving some thoughts) Thanks for this work, it is a very good start and the Javadoc looks clean. To polish, I would be interested to know if Bertil can clarify how "low level" the API needs to be. There is a risk of being a bit close to internal details. For example |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the late reply. Is this docs.zip file a subset of the javadoc that can be found int the following package? Having readers and writers for the different files in a shapefile will be very practical and of great help when implementing higer level abstractions. I checked maven central, but havn't been able to find a release of this dependency yet. Out of curiosity, in the github repository, I still see that there is an implementation jdbc for shapefiles. Is this package still necessary? I believe that most users would prefer to obtain a stream from the FeatureSet and then filter the features with a predicates. I aware that this jdbc implementation does some nice optimizations (like filtering the features based on attributes before reading the geometry), but I'm not sure they are worth the added complexity. Furthermore, there are probably a lot of corner cases that will make this jdbc package difficult to maintain. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding @desruisseaux's question, I agree that the nio API is often difficult to use correctly. When referring to a low-level API, I was thinking about exposing the domain model of the shapefile instead of exposing the domain model of a higher-level abstraction shared between codecs. In this regard, the readHeader and moveToRecord functions would perfectly fit the bill. That being said, I guess some users may specifically want to use the low-level API to access things such as a ChannelDataInput or an InputStream to perform some sort of optimizations or implement additional features. For instance, at some point in Baremaps, we tried to read and write files directly from the network or from S3 without storing local copies of the files. We have been able to do this for a couple of formats, when the InputStream or Channel API were exposed, but we failed when call to these abstractions were only made in private methods. A possibility may be to make things as private as possible in the 'high-level' APIs with a high commitment to API stability. By contrast, the 'low-level' APIs could be more public, while making it clear that there is no commitment to API stability at this level. |
Beta Was this translation helpful? Give feedback.
-
|
Hello. Yes, the attached Javadoc is a subset. It covers only the SIS already allows the use of |
Beta Was this translation helpful? Give feedback.
-
|
@jsorel @desruisseaux I'm currenly working on the creation of vector tiles containing contours and hillshades (all stored as polygons) based on raster DEM and this may be a good opportunity to integrate Apache SIS.
The current solution is based on Terrain Tiles but could be adapted to any tiled dataset. It uses ImageIO to access the elevation data stored in the png files. I initially thought about using gdal2tiles to produces similar raster tiles from datasets such as SRTM or Copernicus DEM, however I'd prefer if the whole pipeline was developped in java. Is this something I could achieve with Apache SIS? From what I understand the GridCoverage does not allows for multiple zoom levels (tile pyramid), but it allows for resampling images and getting the data within a bounding box. Let me know your thoughts on the best way to do this. |
Beta Was this translation helpful? Give feedback.
-
|
Hello Bertil. Apache SIS should be able to use the PNG file directly if it is accompanied by World Files (a More performance can be gained if the data are converted to a TIFF image, preferably COG (Cloud Optimized GeoTIFF) but this is not mandatory. By default, SIS loads the full image in memory. But after the application has been tested to work, SIS can be configured for loading only the tiles needed. Reference: Handle rasters bigger than memory. SIS support pyramids, but it does not appear at the Just for info, there is also an API for creating the contour lines. Please to not hesitate to tell us if we can provide any more indications. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for these pointers, I will make some experiments and come back to you if I have further questions. I will probably start by reading the tiff files of the Copernicus DEM (GLO-30). |
Beta Was this translation helpful? Give feedback.
-
|
@desruisseaux Thanks again for the pointers. A quick update on this. The main issue seems to be related to the sis-epsg jar (non-free), which I cannot add to my dependencies. Is there a way to reproject the rasters from EPSG:4326 to EPSG:3857 without adding the dependency on sis-epsg, perhaps by manually registering the projections? Additionally, I encountered a problem reading cloud-optimized GeoTIFFs stored in MinIO using the S3 integration. I was able to circumvent this issue by using a signed HTTP URL. |
Beta Was this translation helpful? Give feedback.
-
|
Hello Bertil. Thanks for the update and the tip about S3 integration! For EPSG:3857, one way is to create the CRS as below: var crs = CRS.parseWKT(
"""
ProjectedCRS["WGS 84 / Pseudo-Mercator",
BaseGeodCRS["WGS 84",
Datum["World Geodetic System 1984",
Ellipsoid["WGS 84", 6378137.0, 298.257223563]],
Unit["degree", 0.017453292519943295]],
Conversion["Popular Visualisation Pseudo-Mercator",
Method["Popular Visualisation Pseudo Mercator"]],
CS[Cartesian, 2],
Axis["Easting (X)", east],
Axis["Northing (Y)", north],
Unit["metre", 1],
Scope["Certain Web mapping and visualisation applications."],
Id["EPSG", 3857]]
""");Then it should be possible to use that CRS in |
Beta Was this translation helpful? Give feedback.
-
|
@desruisseaux This is great, I will give this a try. |
Beta Was this translation helpful? Give feedback.
-
|
@desruisseaux I'm trying to implement a public class SisTileStore implements TileStore<BufferedImage> {
private final CoordinateReferenceSystem crs;
private final Path path;
public SisTileStore(Path path) {
try {
this.path = path;
this.crs = CRS.fromWKT("""
ProjectedCRS["WGS 84 / Pseudo-Mercator",
BaseGeodCRS["WGS 84",
Datum["World Geodetic System 1984",
Ellipsoid["WGS 84", 6378137.0, 298.257223563]],
Unit["degree", 0.017453292519943295]],
Conversion["Popular Visualisation Pseudo-Mercator",
Method["Popular Visualisation Pseudo Mercator"]],
CS[Cartesian, 2],
Axis["Easting (X)", east],
Axis["Northing (Y)", north],
Unit["metre", 1],
Scope["Certain Web mapping and visualisation applications."],
Id["EPSG", 3857]]
""");
} catch (FactoryException e) {
throw new RuntimeException(e);
}
}
@Override
public BufferedImage read(TileCoord tileCoord) throws TileStoreException {
try (DataStore store = DataStores.open(path)) {
Collection<? extends Resource> allImages = ((Aggregate) store).components();
GridCoverageResource firstImage = (GridCoverageResource) allImages.iterator().next();
firstImage.setLoadingStrategy(RasterLoadingStrategy.AT_GET_TILE_TIME);
var areaOfInterest = new GeneralEnvelope(CommonCRS.WGS84.normalizedGeographic());
areaOfInterest.setRange(0, tileCoord.envelope().getMinX(), tileCoord.envelope().getMaxX());
areaOfInterest.setRange(1, tileCoord.envelope().getMinY(), tileCoord.envelope().getMaxY());
var data = firstImage.read(new GridGeometry(areaOfInterest), null);
var processor = new GridCoverageProcessor();
processor.setInterpolation(Interpolation.BILINEAR);
data = processor.resample(data, crs);
var renderedImage = data.render(null);
var bufferedImage = convertRenderedImage(renderedImage);
return bufferedImage;
} catch (Exception e) {
throw new TileStoreException(e);
}
}
public static BufferedImage convertRenderedImage(RenderedImage img) {
if (img instanceof BufferedImage) {
return (BufferedImage) img;
}
var cm = img.getColorModel();
var width = img.getWidth();
var height = img.getHeight();
var raster = cm
.createCompatibleWritableRaster(width, height);
var isAlphaPremultiplied = cm.isAlphaPremultiplied();
var properties = new Hashtable();
var keys = img.getPropertyNames();
if (keys != null) {
for (int i = 0; i < keys.length; i++) {
properties.put(keys[i], img.getProperty(keys[i]));
}
}
var result = new BufferedImage(cm, raster, isAlphaPremultiplied, properties);
img.copyData(raster);
return result;
}
@Override
public void write(TileCoord tileCoord, BufferedImage blob) throws TileStoreException {
throw new UnsupportedOperationException();
}
@Override
public void delete(TileCoord tileCoord) throws TileStoreException {
throw new UnsupportedOperationException();
}
@Override
public void close() throws Exception {
// Do nothing
}
}The TileStore returns BufferedImages, but they don't have the right scale (under zoomed). I also don't understand why the resutling tiles always have 512x512 pixels. Any idea on how I could make this tilestore behave like gdal_warp? Here are the warp options I currently use: var options = new WarpOptions()
.outputFormat("MEM")
.targetExtent(
envelope.getMinX(), envelope.getMinY(),
envelope.getMaxX(), envelope.getMaxY())
.targetExtentSRS("EPSG:4326")
.targetSize(256, 256)
.targetSRS("EPSG:3857")
.resampling("cubicspline"); |
Beta Was this translation helpful? Give feedback.
-
|
Hello Bertil. There is two However, there is a second Side note: in above code, the |
Beta Was this translation helpful? Give feedback.
-
Hello,
I would like to know if you have any plan to use Apache SIS (https://sis.apache.org/ - https://github.com/apache/sis) ?
There is some overlapping between both projects since SIS is dedicated to Spatial Information System.
Thanks
Beta Was this translation helpful? Give feedback.
All reactions