feature request: save log to local file

[phatpaul]

I would love to see an option to save the logs to a local file (i.e. using VFS), so the logs are actually written to an internal filesystem such as FAT, SPIFFS, LittleFS, etc

How do you retrieve the logging that is written to a local file?

I think it should be left up to the application to provide the method to both write to the file and for the user to retrieve the file(s).

I.e. it could be stored on a SD card and the card removed to retrieve the file.
Or store it to internal flash and download it from an embedded HTTP server. I.e. GET http://192.168.4.1/internalfs/syslog.txt

A modular approach would have this component only provide the core functionality and an API for the application to extend. Then several example projects can demonstrate a few implementations. I could provide an example project that makes the file stored to embedded flash (i.e. FAT) partition and make it available to download via an embedded HTTP server.

Perhaps a useful feature for the core component would be to manage the log file rotation. I.e. how many log files are allowed to be kept, how large a single file can become, and the file naming convention.
I would also like to be able to tee or split (clone) the log to multiple sinks. I.e. it goes to the UART, and is available to stream via one of the supported streaming protocols, and is also written to local file.

[nopnop2002]

Thank you for your valuable opinion.

I would love to see an option to save the logs to a local file (i.e. using VFS),

I believe these features will be provided by new projects such as esp-idf-file-logging rather than additions to this project.

There are a few items to consider.

---

Perhaps a useful feature for the core component would be to manage the log file rotation.

When the file size limit is reached, writing will wrap around to the beginning of the file and continue.
Because strings of any size can be written to the log file, when the file wraps around to the beginning, One of the following actions will occur:

AAAAA<cr><lf> --> 7character
BBBBBBBBBB<cr><lf> --> 12character
CCCCCCCCCCCCCCC<cr> --> 17character
DDDDDDDDDDDDDDDDDD<cr><lf> --> 22character
.
.
.
.
ZZZZZ<cr><lf> --> File limit reached. The next logging will be written to the beginning.
==<cr><lf> --> EOF marking

• Any content up to that point will be discarded.

EEEEEEEEEE<cr><lf>
==<cr><lf> --> EOF marking

• Any content up to that point will be kept.　However, some content will be corrupted.

EEEEEEEEEE<cr><lf> --> 12character
==<cr><lf> --> EOF marking
BBB<cr><lf> ---> remaining 5 characters
CCCCCCCCCCCCCCC<cr><lf> --> 17character
DDDDDDDDDDDDDDDDDD<cr><lf> --> 22character
.
.

---

how many log files are allowed to be kept

What is the purpose of splitting the log files?

[phatpaul]

Thanks for your reply.
I don't think the file rotation should just jump back to the beginning and overwrite the same file. It is not a ring-buffer. That would make it difficult for the user to see where the log starts and ends.
Rather, it should create a new file when a file is full. Also, it should only write an entire line at a time. That will avoid the issue that you described.
But of course this will eventually lead to too many files. So the user could choose how many files to keep.

I.e.. if the user choose to keep the last 4 log files, you would see up to 4 files:
syslog.txt
syslog.txt~1
syslog.txt~2
syslog.txt~3

syslog.txt is always the file written to, then when file size would be exceeded by writing the next log line (in this order):
syslog.txt~3 is deleted.
syslog.txt~2 is renamed to syslog.txt~3
syslog.txt~1 is renamed to syslog.txt~2
syslog.txt is renamed to syslog.txt~1
syslog.txt is created and written to...

[nopnop2002]

I see, that's a good idea.

Do you want to let the user set the maximum size of the log files?
This requires a fair amount of knowledge.
This may be difficult for users.

Or does the component automatically determine it based on the number of files and the free size of the partition?

[phatpaul]

I think the file size should be configurable, with a sensible default i.e. 256kB.

I had not considered what to do if the filesystem is running low on free space. That adds a bit of complexity.
I think the simplest solution could be to add a step during the file rotation that will delete more older logs if the free space is low.

I.e.
#define SYSLOGFILESIZE (256) // kB
when syslog.txt file size would be exceeded by writing the next log line (in this order):
Get filesystem free space;
syslog.txt~3 is always deleted (if exists);
syslog.txt~2 is renamed to syslog.txt~3 or deleted if free space < (SYSLOGFILESIZE * 4);
syslog.txt~1 is renamed to syslog.txt~2 or deleted if free space < (SYSLOGFILESIZE * 3);
syslog.txt is renamed to syslog.txt~1 or deleted if free space < (SYSLOGFILESIZE * 2);
syslog.txt is created and written to...;

[nopnop2002]

Thanks for the specific suggestions.

When building a file system on the built-in SPI Flash, esp-idf supports three file systems: spiffs, fatfs, and littlefs.

It is difficult for users to understand the differences between them.

Which one do you think is the most desirable?

This is Benchmarking with ESP32.
The unit is KB/Sec.

File System	Interface	Write Speed	Read Speed
FATFS	Builtin SPI Flash Memory	46	2625	4096 Byte Sector
SPIFFS	Builtin SPI Flash Memory	17	1034
LITTLEFS	Builtin SPI Flash Memory	51	1678

[nopnop2002]

On a different note, I discovered today that We can use Linux's rsyslogd as a logger.

Log rotation can be left to rsyslogd.

See Using linux rsyslogd as logger in my README.

[phatpaul]

Forwarding the logs over the network to rsyslogd on a Linux server is another great option for people. Nice work!

But it is not an option for every scenario. I.e. I'm am needing to diagnose an infrequent issue that a customer is having on a device in the field (actually on a boat). Adding a Linux server to collect the logs in the field is not an option. Also, this particular issue is involving an unstable network connection, so I don't want to rely on the network to collect logs.
I think the best option for this scenario is to log to the internal memory of the ESP32, and then after the customer has let it run for many hours and experienced the connectivity issues, he can (following my instructions) connect and download the saved logs and send them to me for analysis.

If I could start over, I would choose LittleFS due to it's advertised robustness (performance is much less important). But unfortunately I'm stuck with FAT on thousands of fielded devices. (We have had many instances of a corrupted FAT filesystem.)
Anyway, the choice of filesystem should be left up to the application. The application just needs to provide a VFS path to the logging library of the directory to write the log files to.
An example project that pulls in this component as a library could implement the filesystem so you can still provide an easy example for users that works out-of-the-box.

And I still think the VFS logging option should be part of this same component, so that will enable me to clone the log output to multiple sinks (I.e. logging output can go to all simultanously UART, SSE, and local file, etc).

[nopnop2002]

Anyway, the choice of filesystem should be left up to the application.

I see.
Application prepares the file system and passes the mount point to the initialization function of this library.
For backwards compatibility, you will start file logging in a separate function.

esp_err_t file_logging_init(char * basepath, int number_of_file, size_t size_of_file);

Example of using LittleFS:

	// Mount LittleFS
	char *partition_label = "storage";
	char *mount_point = "/root";
	ESP_ERROR_CHECK(mountLITTLEFS(partition_label, mount_point));

	int16_t write_to_stdout = 0;
#if CONFIG_WRITE_TO_STDOUT
	ESP_LOGI(TAG, "Enable write Logging to STDOUT");
	write_to_stdout = 1;
#endif

	// Start udp logging
        ESP_ERROR_CHECK(udp_logging_init( CONFIG_LOG_UDP_SERVER_IP, CONFIG_LOG_UDP_SERVER_PORT, write_to_stdout );

	// Start file logging. You will see up to 4 files. The maximum file size is 256Kb.
	// /root/syslog.txt
	// /root/syslog.txt~1
	// /root/syslog.txt~2
	// /root/syslog.txt~3
	ESP_ERROR_CHECK(file_logging_init(mount_point, 4, 256*1024));

[mbenabda]

not a bad idea, but i fear that building-in a bunch of features would eventually lead to bringing in dependencies into client project's dependency trees.

This use case makes the point with fs drivers, but besides that there already are esp_http_client and mqtt dependencies brought in so far.

maybe another way to accommodate more features would be to somehow provide access to the ring buffer, and build other companion components around that ? these companion components would bring in the necessary dependencies while allowing feature cherry-picking

[nopnop2002]

If our code become like this, I plan to separate the file log output into a separate repository.
Because it has nothing to do with this repository.

	// Start net logging
	int16_t write_to_stdout = 0;
#if CONFIG_WRITE_TO_STDOUT
	ESP_LOGI(TAG, "Enable write Logging to STDOUT");
	write_to_stdout = 1;
#endif

#if CONFIG_ENABLE_UDP_LOG
	ESP_ERROR_CHECK(udp_logging_init( CONFIG_LOG_UDP_SERVER_IP, CONFIG_LOG_UDP_SERVER_PORT, write_to_stdout ));
#endif // CONFIG_ENABLE_UDP_LOG

#if CONFIG_ENABLE_TCP_LOG
	ESP_ERROR_CHECK(tcp_logging_init( CONFIG_LOG_TCP_SERVER_IP, CONFIG_LOG_TCP_SERVER_PORT, write_to_stdout ));
#endif // CONFIG_ENABLE_TCP_LOG

#if CONFIG_ENABLE_MQTT_LOG
	ESP_ERROR_CHECK(mqtt_logging_init( CONFIG_LOG_MQTT_SERVER_URL, CONFIG_LOG_MQTT_PUB_TOPIC, write_to_stdout ));
#endif // CONFIG_ENABLE_MQTT_LOG

#if CONFIG_ENABLE_HTTP_LOG
	ESP_ERROR_CHECK(http_logging_init( CONFIG_LOG_HTTP_SERVER_URL, write_to_stdout ));
#endif // CONFIG_ENABLE_HTTP_LOG

#if CONFIG_ENABLE_SSE_SERVER_LOG
	ESP_ERROR_CHECK(sse_logging_init( CONFIG_LOG_SSE_LISTEN_PORT, write_to_stdout ));
#endif // CONFIG_ENABLE_SSE_SERVER_LOG

	// Mount LittleFS
	char *partition_label = "storage";
	char *mount_point = "/root";
	ESP_ERROR_CHECK(mountLITTLEFS(partition_label, mount_point));


	// Start file logging. You will see up to 4 files. The maximum file size is 256Kb.
	// /root/syslog.txt
	// /root/syslog.txt~1
	// /root/syslog.txt~2
	// /root/syslog.txt~3
	ESP_ERROR_CHECK(file_logging_init(mount_point, 4, 256*1024));

Just create the following idf_component.yml and your build will be complete.

dependencies:
  nopnop2002/net_logging:
    path: components/net-logging/
    git: https://github.com/nopnop2002/esp-idf-net-logging.git
  nopnop2002/file_logging:
    path: components/file-logging/
    git: https://github.com/nopnop2002/esp-idf-file-logging.git