Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRILL-2362: Profile Mgmt #1750

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

DRILL-2362: Profile Mgmt #1750

wants to merge 2 commits into from

Conversation

kkhatua
Copy link
Contributor

@kkhatua kkhatua commented Apr 15, 2019

NOTE: This PR is a revamp of the work done for DRILL-5270 (PR #1250 and #1654). Those PRs were intended to improve the profile loading time for the WebUI, but did not address the fundamental problem of having too many profiles in the profiles directory.

When Drill is displaying profiles stored on the file system (Local or Distributed), it does so by loading the entire list of .sys.drill files in the profile directory, sorting and deserializing. This can get expensive, since only a single CPU thread does this.
As an example, a directory of 120K profiles, the time to just fetch the list of files alone is about 6 seconds. After that, based on the number of profiles being rendered, the time varies. An average of 30ms is needed to deserialize a standard profile, which translates to an additional 3sec for the rendering of default 100 profiles.

A user reported issue confirms just that:
DRILL-5028 Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS

Additional JIRAs filed ask for managing these profiles
DRILL-2362 Drill should manage Query Profiling archiving
DRILL-2861 enhance drill profile file management

This PR brings the following enhancements to ensure that profiles are better managed.

  1. All profiles are now written into directory partitions based on the timestamp of the queries. By default, they will be written to <profileDir>/yyyy/mm/dd
  2. During startup, the Drillbit checks if there are any profiles already in the root directory, and will move those into their correct partitions. For sake of performance, this is restricted to the 10000 (configurable) most recent profiles.
  3. If the profile directory is on a distributed filesystem and there are un-indexed/partitioned profiles (see # 2 above), the Drillbits use Zookeeper to synchronize on who will do the indexing/partitioning.
  4. When profiles are listed in the WebUI, the WebServer will only explore and fetch profiles from the most recent partitions, rather than explore the entire partitioned space.
  5. When an individual profile is accessed based on its query ID, Drill is try to extrapolate the time range of when the query was submitted. This time-range is used to infer the possible partitions. These partitions are then explored to find the profile. This is essentially reverse engineering the process of query ID generation.
    Reference: https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/user/UserWorker.java#L67
  6. We also append list of parent directories. This is done because if the partition format has been made more granular, we don't repartition the profiles. So Drill will miss profiles in directories that were formerly leaf directories.
    e.g. moving from yyyy/mm/dd to yyyy/mm/dd/hh
  7. The WebUI often accesses the profile twice instead of once, because, in addition to the WebServer making a call to the JSON file for rendering, the fragment Gantt chart's javascript also makes an independent call to the WebServer. To mitigate this, Drill will leverage Guava cache to store the 200 most recently accessed profiles. This means that when a user lists profiles and then chooses to visit one, we can take advantage of the cache to avoid making 3 calls to the filesystem.
  8. A separate directory - diorama , also exists in the root of the profile directory. The purpose of this is to allow users to dump external profiles that can be then rendered and visualized in the WebUI. Currently, the profile needs to be dumped manually and cannot be done via the WebUI. This could be a future enhancement,

@kkhatua
Copy link
Contributor Author

kkhatua commented Apr 15, 2019

@arina-ielchiieva I'm looking for a suggestion on how to manage existing profiles.
Currently, during startup, we automatically index any profiles in the root of the profiles dir into their respective location. As a default, I have set it at 1000, and the Drillbits syncrhonize if the profile dir is on a DFS.
However, I am not sure if we should automatically index any new profile that has been copy-pasted into the root dir. For e.g., we might get a profile from a JIRA and would like to view it. Should we leave it there (and try to render it) or should we index it ASAP ?

@kkhatua kkhatua self-assigned this May 17, 2019
@kkhatua kkhatua marked this pull request as ready for review May 17, 2019 21:52
@kkhatua
Copy link
Contributor Author

kkhatua commented May 17, 2019

@arina-ielchiieva could you please review this PR ?

@cgivre cgivre added the enhancement PRs that add a new functionality to Drill label Jan 10, 2021
@jnturton
Copy link
Contributor

jnturton commented Sep 23, 2022

Hi @kkhatua! Thank you for this contribution, I'd like to help to move it forward.

However, I am not sure if we should automatically index any new profile that has been copy-pasted into the root dir. For e.g., we might get a profile from a JIRA and would like to view it. Should we leave it there (and try to render it) or should we index it ASAP ?

I share your concern here. I think we should consider having Drill only write new profiles to partitioned directories. Any partitioning of historical profiles can be done externally by admins, in my opinion, and we can add examples of "housekeeping" scripts for doing that to the Drill documentation.

Would you like to do any of the following?

  • Resume the work on this PR, I volunteer to be a reviewer.
  • Receive a PR from me to your fork here that's rebased and has some changes I think we want.
  • You're too busy now, so I'll pull your commits here into a new branch of my own and open a new PR.

Thanks
James

@jnturton
Copy link
Contributor

Or, if we do want to keep this built in ability to partition existing profiles, perhaps we should have it launched from a button on the Profiles page in the web UI instead of on Drillbit startup? That would remove the complication of which Drillbit does the work and the worries of slowing down startup or partitioning profiles that nobody wanted partitioned.

@cgivre cgivre added the revisit label Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement PRs that add a new functionality to Drill revisit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants