Major problems with “loading data into memory” – long delays, strange data styling effects and even crashes #2614
Comments
|
Literally everything you list as symptom points to you running out of memory (and we've discussed this before). There is no reason to believe that the changes in 20.1 would improve this, actually pre-loading the NSI will likely make it worse. And as your last screenshot shows, it isn't the amount of OSM data that is causing the issue (but you are using 60% of the total available memory there with no obvious cause). Things I would suggest trying:
further check for any settings/functions on your device that could remove vespucci from memory that are available, in general it should be -difficult- to get the app to completely restart (loading the state in to memory essentially only happens then) and should really only be the case if you manually completely exited it. Check that you don't have the "Don't keep activities" option turned on in the Android developer options. I would further note that both state files seem to be excessively large in that last screenshot, resetting both the OSM data and tasks now and then after editing and uploading might help with that. PS: the 0.5 GB maximum memory is the same on all modern Android devices, so there is nothing special about the memory budget for your device, aka mine is exactly the same. |
|
Is this crash dump from ~2 weeks ago from you? Its an A33, and the crash is happening on parsing the preset after changing the configuration. |
Yes, it's mine. I only had such a crash like this one time. I think there was no syntax error in my presets or datastyling when it happened (as far as I can remember). And as I wrote, I deactivated all my files to check something. I don't remember any further details. Except that I was able to restart Vespucci after the crash and there were no particular problems (as far as I can remember). Otherwise I would probably have written a GitHub issue. |
|
The point is not that it crashed as a result of the particular action, but that it crashed as the result of an out of memory error. Once you are (so) low on memory such an error can happen essentially anywhere but it underlines that the likely cause of your issues is not enough memory/something using it up very fast. |
|
Pro memoria: the NSI seems to currently require ~50MB of memory. |
Thank you for all the hints ... NSI:To understand it correctly: When "Loading data into memory" appears, is it loading or updating the NSI from the network? That would explain why it is so bad when the network reception is poor (e.g. in dense forest). Is it loading/updating anything else from the network (e.g. OSM notes or Osmose warnings)? I had "Enable name suggestions" activated + now deactivated (I assume that is what the NSI is used for). "Enable name suggestion prese.." was already deactivated. Will this stop pre-loading the NSI on startup? GPX tracks:I have one GPX layer with a self-made GPX file which only contains 24 way points – it has 32 KB. (The debug info says: "GPX 6 GPX true content://com.android.providers.downloads.document/msf%3A1000011300" – but the file name of the GPX file is different). I could delete that layer if it is helpful. In rare cases I used the GPX recording for a short time, but after that I always used "Delete GPX track and its waypoints". Now, there is no track. Effect of any additionally loaded presetsI can try to check next time, if deactivating all my (3) custom presets and only use your current "Simonpoole last release" preset (or better only the build in preset?). Or what do you have in mind? I assume it only loads the activated presets during startup, is that correct? Normally my 3 presets are:
Settings/functions on your device that could remove Vespucci from memoryAs I wrote, I had already checked everything that came to mind, but this is really a bad thing with Android or the manufacturer's own functions for energy saving and so on ... Some Android settings I use:
Android developer optionsI always had "Don't keep activities" option turned off. But I have to admit that I completely forgot to check the Developer Options. Very stupid ... I had turned them on some time ago (when the problems described were already there) to see if anything could help improve things. I don't remember exactly when I activated it And stupidly, I had this still activated:
I have now completely turned off the Developer Options for further testing. (Small side note: although I studied computer science in the late 80s/early 90s, I never worked as a developer or programmer as my main job ... but as a project manager, among other things.) Memory and state fileRunning out of memory: What would a "normal" amount of used memory? And what are the potential candidates for using up the rest of the memory in a fast way (if I don't download new OSM data and all presets/datastyling is already loaded)? Could it have anything to do with Javascript items (I use a lot of them, but most of them only with small and simple Javascript code for setting or removing tags – only my traffic_sign Javascript items are larger and more complex; see infos about presets above)? But could they be responsible for memory not being released properly, for example if they are used frequently? State file: Bug state file: Sometimes, I perhaps had downloaded a larger area of OSM data (I try to avoid that), but of course I always use "Clear and download current view" after an upload and if starting with a new area. Or should I also use "Clear data" from time to time (I mean does it clear more data)? Resetting tasks:
Maybe I need a few more days of testing, especially with developer options disabled (and NSI disabled)... And maybe just use your or the internal presets if there are any problems - I don't think there's much I can change/improve to reduce memory usage for example. Then I'll see what happens. I will write then again. I'm not quite sure if the weather in the next few days will be good enough for a longer mapping tour. But it would be helpful if you could answer some (or even all?) of my questions in the meantime. Thank you very much! |
Ok, I understand this connection. It certainly fits the situation – it is very likely that I also changed the presets/data styling when there was a problem, to see if it helped. But I can't remember if I had restarted Vespucci BEFOREHAND, for example, and if a lot of OSM data had been loaded and changes had not yet been uploaded, etc. But I'm pretty sure I was busy mapping... |
The problem is not just the sum of available memory but if there are blocks of the required size (so very fragmented memory may still lead to an OOM even if you are asking for less than total free memory). But I don't think you are doing anything that "in general" can't work with the memory you have available just that you are triggering something in a way that leads to an issue. The default preset uses around 20MB and the NSI around 50MB as an indication. Taken as is, that shouldn't be an issue, however presets contain images that need to be converted to Android bitmaps for display (that will increase their size) and I suspect that this might be one of the causes for the issue. Tasks include all Variants, that is Notes, Maproulette Tasks, OSMOSE, and Todos. Resetting will definitely delete anything held in memory. However if you simply manually run "prune" the Todos will not be affected. The state files contain a serialized version of the on the one hand OSM data plus undo information and on the other hand the same for tasks. There are further files holding editing state and per layer information, but they are typically not critical. And yes "loading data into memory" is displayed while the state files are being read. |
|
Hello ... thank you. Some more light in the dark. Just for info:
First experiences yesterday (with Developer Options turned off completely + without NSI in Vespucci); I also choosed "prune" in the task layer and tried to use the recent apps list if possible to return to Vespucci when switching apps.
Then a strange thing (never had this before):
Then I did a device restart to be sure ... Then also "Close all apps" in the recent apps list.
Today (at home, Wifi connection):
What I basically don't understand: (And why did it happen also if I used the flight mode without any connection – didn't Vespucci notice this?) And a question about the suspected OOM problems: Last questions:
I'll be watching what happens more closely over the next few days. If there's anything else I should look out for, please let me know. |
|
The 1GB native heap is obviously very large, particularly considering that the app doesn't actually explicitly use it, so those allocations are either made by Android or a library we use, but I'm not aware of any that we use that does that. My suspicion is that that might be image/bitmap related. Wild guess: SVG icons? The other thing that comes to mind wrt general memory usage is data styling, I can't really see how this could amount to any noticeable amount, but all available data styles are parsed on start up and held in memory. While behaviour of the app is naturally impacted by bad network connectivity, it shouldn't have any bearing on the issues you are seeing. PS: as a tendency I would increase the on-device tile cache size from the 100MB you currently have configured if you are in areas with bad connectivity. |
|
Hello ... I haven't done much mapping in the last 3 days, so I haven't been able to test much, just a little in a day. But first I did some more in-depth research into possible Android and Samsung system settings that could play a role. And I've known for a while that Samsung is known for having a pretty aggressive battery management system, which includes terminating foreground and background processes. It is quite well documented on https://dontkillmyapp.com/samsung, perhaps you know this site (and the app "DontKillMyApp"). I researched all of this a few months ago, but it's really hard for an end user to decide which settings really help with a problem and what they actually do. Anything but nice... But what I've now found out and changed, which you might also be interested to know (some if this is Samsung specific):
Maybe interesting for you, or you already know that (?):
And one of my main problems with everything is that Vespucci seems to be restarting over and over again and this start-up process takes so long or too long, sometimes leading to ANR conditions. If that didn't happen anymore, it would be a big improvement (or even solve all problems?). First experiences with the changed settings (especially on one day, about 3 hours):
Here are some screenshots when the ANR dialog appeared:
Here is a final screenshot with the state file > 3 MB: Otherwise, I have now increased the tile cache to 300 MB (OK?) and set "Max. number of download threads" from 1 to 2 (what is the default value?). And regarding the data styling and SVG icons (which I use in data styling as well as in my presets): I could possibly replace them with PNGs with 48 x 48 px as a test, even if that would be a bit of work. And the PNGs really do look much worse ... once you have seen the difference! It would be great if you could use JPGs/PNGs in higher (e.g. double) resolution for high-resolution screens, similar to what is possible in web design. And if scaling the pixel graphics would then be necessary: perhaps it is less memory/processing intensive than SVG processing? Although I actually think that the graphics routines for SVG rendering should be very efficient, but you never know. Off topic: I also really miss an option to specify the icon size for data styling (as is possible for JOSM) or the proportional rendering of SVG graphics - currently non-square graphics are distorted to a square size... So much for today ... |
Android does not have any native SVG support, so the only thing we do is render the SVG to a bitmap on device. The way things are done now we allocate a bitmap the size of the SVG render to that and then scale that for actual display. This could very well use a lot of memory if the SVG dimension are large. I'm testing a modification that allocates a smaller bitmap that might help with this potential issue. |
|
For your infomation: I haven’t been able to do any real “stress tests” with longer mapping tours in the last few days. That's why I haven't written anything else. However, I once had the ANR dialog in a loop again (3 days ago, I was out and about). "Clear data" in Vespucci, restarting Vespucci (small State file after that) and (before) "Clear memory" in the Android settings didn't help either. Around 10 minutes later (at the same location, only around 50 m away) it started working again. And then for the next 3 hours there was no ANR dialog - despite using other apps, a larger map section was loaded etc. I just can't see any regularity as to when it occurs and when it doesn't. Once in the evening (at home) I had the phenomenon again with the data styling reset to minimal (black lines etc.), but no ANR dialog! But a fairly large map section was loaded (state file 2.9 MB) ... And I had switched back and forth a lot between different apps, so maybe the Android memory management? And another piece of information: So far I haven't had any ANR dialog, and the "Loading presets" process seems to be 1-2 seconds faster (usually about 6 seconds). Of course the preset and datastyling (zip) files are also a bit smaller ... |
The rendering of all icons is on demand, so isn't happening while the preset is being read, but for example while the first map is being displayed. |
|
I have some new information (and some new insights). First because of replacing all SVGs with PNGs (in my custom data styling and the presets):
New insights (and assumptions) regarding multiple custom data stylings:This might even be/have been the main problem besides the memory requirements of the rendered SVGs (which by the way are all created in the size 580 x580 px, a few maybe in 600 x 600 px, but not larger). What happened:
Here is a debug info screenshot (after the import of 4 new custom data stylings):
What I then discovered when reconfiguring Vespucci (v20.1Beta4):
Since then, no major problems (apart from the frequent app restarts). So far, no ANR state, no crash, no minimal data styling. But no very large "stress tests" either... Main question/assumption after all this:I now suspect that ALL imported data stylings are loaded into memory when the app starts and are kept there (or when a new one is imported) – at least the styling rules (If the PNGs used in data styling by iconPath="XY" are perhaps also only loaded when needed for the map, like you explained for the display of the icons in the presets?). And not just the currently activated one. I wasn't really aware of that until now, but it would (perhaps) explain a lot. So the most important question is: is that the case? It would explain why I had such high memory consumption the whole time and Vespucci became unusable after adding 4 new data stylings, as described above. (Although it is not 100% clear to me why a data styling that is 12 MB unzipped - and only with PNGs - might lead to a memory requirement of 35 or 44 MB (see above), and whether there are opportunities for optimization. But maybe that is not the most important thing. The big problem is that you can add custom data stylings with "Import data style", but you cannot remove what you have added (there is no "Remove custom data styling" as for presets). I had already noted this in 2022 in #1874 (that a similar interface for data stylings as for Manage Presets would be very desirable), but nothing has changed since then. I will try to create a new issue for it soon, because I think that would be very important. But maybe I am the only Vespucci user who uses custom data styling (or such extensive ones) and who then encounters this problem? And because that's the case and I hadn't completely reset Vespucci in a long time (reinstall or clear app data), I had a total of 5 custom data stylings for a long time (most with smaller XMLs than my current version, but it would perhaps explain the high memory consumption and OOM situations if the styling rules of all XMLs are kept in memory and these also require significant memory. Plus the SVG problem...) As long as there is no "Remove custom data styling" function, I will make sure not to import more than my 2 styling variants, i.e. not to assign new profile names. I would of course also like to test with SVGs again as soon as your SVG scaling/rendering change is published (20.1Beta5?), and assume that I can then import an SVG data styling version with the same profile name as the PNG only version. And that I can then change it back again (PNG only) without this resulting in additional RAM usage (perhaps space usage in the internal memory, if SVGs once imported remain there? That wouldn't be a big problem... I don't know if the DataStyling is unzipped when imported and how the handling of graphic files that are no longer used looks like). Basically, it would be very helpful for me if you could publish a short information about what exactly happens when the app starts and in what order (and what is permanently stored in the RAM or removed again by various commands and functions in the app). At the moment I can only guess - e.g. because there is no significant delay when you change the data styling (= all stored in memory?), but a clearly noticeable delay when you have changed the preset selection and exit the settings (I suspect that the selected presets are then loaded into memory and deactivated presets are removed from memory, so it doesn't matter whether the list of presets contains many - deactivated - presets, unlike with data styling). Off topic: By the way, I have just created another issue – #2632 – regarding preset activation/deactivation (or loading a new version), because it would be necessary for the validator to be reinitialized every time the presets are changed. If there are changes in the presets (e.g. if you have made validator check keys optional in preset items), the validator engine will not immediately recognize this; you must first change something in the validator check keys to trigger a re-initialization. |
|
Beta 4 contained the changes to SVG rendering https://github.com/MarcusWolschon/osmeditor4android/releases/tag/20.1.0.4 (a 580px X 580 px image corresponds roughly to a 1.3MB bitmap if not scaled down to the target icon size). In general it isn't possible to definitely document when things are loaded and not, as nearly all of it happens asynchronously to the main UI thread running. More so, garbage collection of objects that have no active references any more is controlled by the JVM, there is no guarantee that memory used by objects that are no longer referenced is immediate (which is likely what you are seeing). All available data styles are currently held in memory that that shouldn't really be an issue, as the classes should be small, but potentially there might be some references retained in the icon caches when you are switching back and forward which potentially could be improved. |
|
Can this issue be closed? |
A good question ... difficult to answer. There are still some things I would like to understand better. And some questions are unanswered. First of all: the problem has been significantly reduced for me since I stopped using SVGs (instead, PNGs in 48x48px) and only used 2 custom data styling files. I can now map much more relaxed again... Since then, practically no more ANR states or these "minimal data styling" effects, possibly caused by OOM states. Problems/open questions that remain include:
|
Except for "cold starts" aka the app was previously terminated and removed from memory, switching to the app should be nearly instantaneous. Reading the saved data state is typically only an issue if the system needs to garbage collect while doing so. There is a specific aspect that likely mainly effects you, that the data styles are currently not read asynchronously on start up, so if you have a lot of them that might be an issue (I'm refactoring the relevant code so there might be some improvements there, however the styles need to be available before the data is rendered so that will always a bit of a limit on things).
In general the app never restarts for me (same Samsung OS Android version), aka I have to explicitly exit completely to make that happen. Things you can check are for example the background battery usage setting for the app. The app has always fulfilled any google set policy restrictions etc for services/background use and this is no different, the new policy requires apps to explicitly require a subject matter specific permission for foreground services (instead of an unspecific one) but that is just about being able to run the service in the first place (which just affects GPX recording BTW). 20.2 has an option to enable "hardware" rendering which allows bitmaps etc to be stored in memory outside of the Java VMs heap which may or may not help with your issue, but at least in principle makes more memory available to the app.
Simply adding additional data styles should not use any significant amount of memory (I would suspect perhaps 1kB per feature entry so, ~0.5-1MB per style), switching between them if they have many custom icons will cause more use because of caching of icons etc, till the system re-runs garbage collection.
See my comment on the relevant issue.
The map icons are 20x20 dip iirc. PNG are scaled down to that size, so likely SVG rendering to that size will still be better.
See Add a function to remove custom data stylings #2631
Yes. |
|
Closing as stale and 20.2 is more or less released at this point. |
Vespucci Version
20.1.0 Beta3
Download source
Google Play Store
Device (Manufacturer and Model)
Samsung A33
Android Version
14
Behaviour/Symptoms
I have been experiencing a very annoying issue with very long delays with the message "Loading data into memory" after switching apps or going to and from the home screen for a while (already under Android 13 and Vespucci v19.x, and now under 20.0.x and 20.1.0 Beta3). My first screenshot with the problem is from 2023-08-09. It frequently makes mapping difficult or even impossible when you are on the move. I also tried to use the recent apps list only to go back to Vespucci, but I think (not 100% sure) that this does not help to solve the problem.
Describing and reproducing the problem is not easy, as it does not occur in every case, but very frequently and practically always during a mapping tour - and often it does not stop or I do not know how to stop it (I have tried everything possible). And often it becomes a kind of loop and you can no longer carry on normally. And it always costs a lot of time and energy.
I was hoping that the new fixes in v21.1.0 regarding app launch might resolve this, but unfortunately not. That's why I'm writing this (a little longer …) issue now. Perhaps it's also a special Samsung or a general Android problem? Don't know.
There are several effects of varying severity that, in my opinion, are all related to one another. I will try to describe it (from less severe to worse effects):
Example sequence 1 (without “worst case”, only a long delay up to 20 s, no ANR dialog):
Example sequence 2 (with ANR dialog):
Example sequence 3 (with data styling reset and delay > 60 s):
amenity=fast_food) called up in the editor for editingcontact:phone+contact:website), then the URL marked and copiedcontact:phoneandcontact:websiteare gone (this problem seems to be gone with the new experimental settingUse "new task" mode for property editor– I have not been able to check it extensively yet, in connection with this problem).amenity=fast_foodnode.Just some suspicions/thoughts:
Use "new task" mode for property editor– not sure.For information:
I have these questions too (among others), maybe you could shed some light on them?
When the message "Loading data into memory" appears, what is actually being loaded? Only the last saved state and all of the active presets, data styling (= local files)? Or is another data transfer over the network required like tiles or OpenStreetMap Notes or something similar (since, presumably, a data transfer seems to take place before Vespucci can be used once more; at the very least, transfer arrows are visible on the icon in the Android header bar).
And of course: can these issues be solved? Would really be great!!!
How to recreate
See notes above. I haven't found a reliable way to reproduce it yet, all I can say is that it happens to me pretty much every day when I'm mapping.
Some screenshots
Black screen at startup (for some seconds):
Minimal data styling (in rare cases):
Some debug infos:
Crash dump submitted (no or yes + date)
Yes, several times (I don't remember the dates; I have one crash screenshot from 2024-03-26). I remember adding a text like "Problems after loading data into memory" at least once.
The text was updated successfully, but these errors were encountered: