Adding support for Library Panels CRUD (#135)

* Adding support for library elements.

PR for #67

ChangeLog:
  - Adding support for Library Panels CRUD
  - Refactoring Filters to be more generic.
  - Introducing a --use-filters.  To only operated on monitored folders
    for the folder CRUD.
  - Adding integration test
  -  Adding -s -w build flags

.goreleaser.yml

@@ -10,7 +10,7 @@ builds:
   - env:
       - CGO_ENABLED=0
     binary: gdg
-    ldflags: -X github.com/esnet/gdg/version.GitCommit={{ .Commit }} -X github.com/esnet/gdg/version.BuildDate={{ .Date }}  -X github.com/esnet/gdg/version.Version={{ .Tag }}
+    ldflags: -s -w -X github.com/esnet/gdg/version.GitCommit={{ .Commit }} -X github.com/esnet/gdg/version.BuildDate={{ .Date }}  -X github.com/esnet/gdg/version.Version={{ .Tag }}
     goos:
       - linux
       - windows

Makefile

@@ -27,7 +27,7 @@ authors:
 
 
 linux: clean 
-	env GOOS='linux' GOARCH='amd64' go build -ldflags "-X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}" -o bin/${BIN_NAME}_linux
+	env GOOS='linux' GOARCH='amd64' go build -ldflags "-s -w -X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}" -o bin/${BIN_NAME}_linux
 
 help:
 	@echo 'Management commands for gdg:'
@@ -48,12 +48,12 @@ help:
 build:
 	@echo "building ${BIN_NAME} ${VERSION}"
 	@echo "GOPATH=${GOPATH}"
-	go build -ldflags "-X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}" -o bin/${BIN_NAME}
+	go build -ldflags "-s -w -X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}" -o bin/${BIN_NAME}
 
 install:
 	@echo "installing ${BIN_NAME} ${VERSION}"
 	@echo "GOPATH=${GOPATH}"
-	go install -ldflags "-X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}"
+	go install -ldflags "-s -w -X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}"
 	mv ${GOPATH}/bin/gdg ${GOPATH}/bin/${BIN_NAME}
 
 get-deps:
@@ -62,7 +62,7 @@ get-deps:
 build-alpine:
 	@echo "building ${BIN_NAME} ${VERSION}"
 	@echo "GOPATH=${GOPATH}"
-	go build -ldflags '-w -linkmode external -extldflags "-static" -X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}' -o bin/${BIN_NAME}
+	go build -ldflags '-s -w -w -linkmode external -extldflags "-static" -X github.com/esnet/gdg/version.GitCommit=${GIT_COMMIT}${GIT_DIRTY} -X github.com/esnet/gdg/version.BuildDate=${BUILD_DATE}' -o bin/${BIN_NAME}
 
 package:
 	@echo "building image ${BIN_NAME} ${VERSION} $(GIT_COMMIT)"

README.md

@@ -13,38 +13,7 @@ Please find the pretty documentation [here](https://software.es.net/gdg/docs/usa
 
 ## Breaking Changes
 
-ChangeLog: 0.4.X
-
-This is a major change from the previous one, I'll likely cut the 1.x soon and start following the more typical Semver conventions.  
-
-### API SDK Changes:
-
-I have been trying to find a proper library to use so I'm not re-writing and reinventing the wheel so to speak. 
-
-For reference, here are all the current "active" (active can be a relative term for some of these project) development I'm aware of.
-
-  - https://github.com/grafana-tools/sdk Initial version of GDG was based on this project.  It mostly works but getting any PRS accepted can be tedious and it's needs some help.  
-  - https://github.com/grafana/grafana-api-golang-client Owned by the Grafana Org which is nice, but it has a slightly different goal.  It's primary goal is to support the terraform provider for Grafana.  I also found some endpoints missing very early on.  So decided not to go with it.
-  - Swagger Based:  There's a branch that I've been keeping an eye on.  https://github.com/grafana/grafana-api-golang-client/tree/papagian/generate-client-from-swagger which makes an effort to generate code based on the swagger manifest that's available from Grafana.  It's a mostly automated code that pulls data from the [Schema](https://github.com/grafana/grafana/blob/main/public/api-merged.json) and generates the underlying code.  It hasn't had much traction of late so I ended up forking the project currently available [here](https://github.com/esnet/grafana-swagger-api-golang)
-
-Final Choice:
-
-Although the Swagger/OpenAPI based version is not great, I've even ran into a few issues where the documented response 
-does not match the result, it's a lot more encompassing and allows further development without being as limited on upstream changes.
-
-## DataModel Changes
-
-I've tried to utilize mostly the same endpoints to recreate the same behavior for all the various entities, but there 
-is are some changes.  For most use cases this shouldn't matter.  But you have been officially warned.
-
-## Cloud Support
-
-The previous abstraction library used to provide S3, GS, SFTP has limited activity and introduced some security vulnerabilities.  0.4.X also 
-changes some of the cloud behavior.  It relies on the system authentication rather than having the auth in the config file.
-
-Please see the related docs on how to configure your environment.
-
-As the Stow library was removed, SFTP has been dropped.  The list of current supported cloud providers are: S3, GS, Azure.
+Release 0.4 has some breaking changes, please read the release (notes)[https://software.es.net/gdg/docs/releases/gdg_0.4.0/]
 
 ## Quickstart 
 

api/alertnotifications.go

@@ -16,7 +16,6 @@ import (
 // AlertNotificationsApi Contract definition
 // Deprecated: Marked as Deprecated as of Grafana 9.0, Moving to ContactPoints is recommended
 type AlertNotificationsApi interface {
-	//AlertNotifications
 	ListAlertNotifications() []*models.AlertNotification
 	ImportAlertNotifications() []string
 	ExportAlertNotifications() []string

api/common.go

@@ -3,7 +3,6 @@ package api
 import (
 	"fmt"
 	"github.com/esnet/gdg/config"
-	"github.com/esnet/grafana-swagger-api-golang/goclient/models"
 	"os"
 	"path/filepath"
 	"strings"
@@ -14,8 +13,10 @@ import (
 )
 
 var (
-	DefaultFolderName = "General"
-	DefaultFolderId   = int64(0)
+	DefaultFolderName   = "General"
+	DefaultFolderId     = int64(0)
+	searchTypeDashboard = "dash-db"
+	searchTypeFolder    = "dash-folder"
 )
 
 func GetSlug(title string) string {
@@ -56,12 +57,3 @@ func buildResourcePath(folderName string, resourceType config.ResourceType) stri
 	return v
 
 }
-
-// getFolderNameIDMap helper function to build a mapping for name to folderID
-func getFolderNameIDMap(folders []*models.FolderSearchHit) map[string]int64 {
-	var folderMap = make(map[string]int64)
-	for _, folder := range folders {
-		folderMap[folder.Title] = folder.ID
-	}
-	return folderMap
-}

api/contract.go

@@ -20,6 +20,7 @@ type ApiService interface {
 	AlertNotificationsApi
 	UsersApi
 	FoldersApi
+	LibraryElementsApi
 
 	//MetaData
 	GetServerInfo() map[string]interface{}

api/dashboards.go

@@ -3,6 +3,7 @@ package api
 import (
 	"encoding/json"
 	"fmt"
+	"github.com/esnet/gdg/api/filters"
 	"github.com/esnet/gdg/config"
 	gapi "github.com/esnet/grafana-swagger-api-golang"
 	"github.com/esnet/grafana-swagger-api-golang/goclient/client/dashboards"
@@ -14,6 +15,7 @@ import (
 	"golang.org/x/exp/slices"
 	"os"
 	"path/filepath"
+	"regexp"
 	"strings"
 
 	"github.com/esnet/gdg/apphelpers"
@@ -26,15 +28,88 @@ import (
 
 // DashboardsApi Contract definition
 type DashboardsApi interface {
-	ListDashboards(filter Filter) []*models.Hit
-	ImportDashboards(filter Filter) []string
-	ExportDashboards(filter Filter)
-	DeleteAllDashboards(filter Filter) []string
+	ListDashboards(filter filters.Filter) []*models.Hit
+	ImportDashboards(filter filters.Filter) []string
+	ExportDashboards(filter filters.Filter)
+	DeleteAllDashboards(filter filters.Filter) []string
+}
+
+func NewDashboardFilter(entries ...string) filters.Filter {
+	if len(entries) != 3 {
+		log.Fatal("Unable to create a valid Dashboard Filter, aborting.")
+	}
+	folderFilter := entries[0]
+	dashboardFilter := entries[1]
+	tagsFilter := entries[2]
+
+	filterObj := filters.NewBaseFilter()
+	filterObj.AddFilter(filters.FolderFilter, folderFilter)
+	filterObj.AddFilter(filters.DashFilter, dashboardFilter)
+	filterObj.AddFilter(filters.TagsFilter, tagsFilter)
+	quoteRegex, _ := regexp.Compile("['\"]+")
+	filterObj.AddRegex(filters.TagsFilter, quoteRegex)
+	filterObj.AddRegex(filters.FolderFilter, quoteRegex)
+	//Add Folder Validation
+	filterObj.AddValidation(filters.FolderFilter, func(i interface{}) bool {
+		val, ok := i.(map[filters.FilterType]string)
+		if !ok {
+			return ok
+		}
+		//Check folder
+		if folderFilter, ok = val[filters.FolderFilter]; ok {
+			if filterObj.GetFilter(filters.FolderFilter) == "" {
+				return true
+			} else {
+				return folderFilter == filterObj.GetFilter(filters.FolderFilter)
+			}
+		} else {
+			return true
+		}
+	})
+
+	//Add Tag Validation
+	filterObj.AddValidation(filters.TagsFilter, func(i interface{}) bool {
+		val, ok := i.(map[filters.FilterType]string)
+		if !ok {
+			return ok
+		}
+
+		//Check Tags
+		if tagsFilter, ok = val[filters.TagsFilter]; ok {
+			if filterObj.GetFilter(filters.TagsFilter) == "" {
+				return true
+			}
+			return tagsFilter == filterObj.GetFilter(filters.TagsFilter)
+		} else {
+			return true
+		}
+		//Check Dashboard
+
+	})
+	//Add DashValidation
+	filterObj.AddValidation(filters.DashFilter, func(i interface{}) bool {
+		val, ok := i.(map[filters.FilterType]string)
+		if !ok {
+			return ok
+		}
+
+		if dashboardFilter, ok = val[filters.DashFilter]; ok {
+			if filterObj.GetFilter(filters.DashFilter) == "" {
+				return true
+			}
+			return dashboardFilter == filterObj.GetFilter(filters.DashFilter)
+		} else {
+			return true
+		}
+
+	})
+
+	return filterObj
 }
 
 // ListDashboards List all dashboards optionally filtered by folder name. If folderFilters
 // is blank, defaults to the configured Monitored folders
-func (s *DashNGoImpl) ListDashboards(filters Filter) []*models.Hit {
+func (s *DashNGoImpl) ListDashboards(filterReq filters.Filter) []*models.Hit {
 
 	var orgsPayload []*models.OrgDTO
 	orgList, err := s.client.Orgs.SearchOrgs(orgs.NewSearchOrgsParams(), s.getAdminAuth())
@@ -63,8 +138,8 @@ func (s *DashNGoImpl) ListDashboards(filters Filter) []*models.Hit {
 	}
 
 	// Fallback on defaults
-	if filters == nil {
-		filters = &DashboardFilter{}
+	if filterReq == nil {
+		filterReq = NewDashboardFilter("", "", "")
 	}
 
 	var boardsList = make([]*models.Hit, 0)
@@ -75,15 +150,15 @@ func (s *DashNGoImpl) ListDashboards(filters Filter) []*models.Hit {
 
 	var tagsParams = make([]string, 0)
 	if !apphelpers.GetCtxDefaultGrafanaConfig().GetFilterOverrides().IgnoreDashboardFilters {
-		tagsParams = append(tagsParams, filters.GetTags()...)
+		tagsParams = append(tagsParams, filterReq.GetEntity(filters.TagsFilter)...)
 	}
 
 	for {
 		searchParams := search.NewSearchParams()
 		searchParams.Tag = tagsParams
 		searchParams.Limit = gapi.ToPtr(int64(limit))
 		searchParams.Page = gapi.ToPtr(int64(page))
-		searchParams.Type = gapi.ToPtr("dash-db")
+		searchParams.Type = gapi.ToPtr(searchTypeDashboard)
 
 		pageBoardLinks, err := s.client.Search.Search(searchParams, s.getAuth())
 		if err != nil {
@@ -96,7 +171,7 @@ func (s *DashNGoImpl) ListDashboards(filters Filter) []*models.Hit {
 		page += 1
 	}
 
-	folderFilters := filters.GetFolders()
+	folderFilters := filterReq.GetEntity(filters.FolderFilter)
 	var validFolder bool
 	var validUid bool
 	for _, link := range boardLinks {
@@ -113,8 +188,9 @@ func (s *DashNGoImpl) ListDashboards(filters Filter) []*models.Hit {
 			continue
 		}
 		link.Slug = updateSlug(link.URI)
-		validUid = filters.GetFilter("DashFilter") == "" || link.Slug == filters.GetFilter("DashFilter")
+		validUid = filterReq.GetFilter(filters.DashFilter) == "" || link.Slug == filterReq.GetFilter(filters.DashFilter)
 		if link.FolderID == 0 {
+
 			link.FolderTitle = DefaultFolderName
 		}
 
@@ -128,7 +204,7 @@ func (s *DashNGoImpl) ListDashboards(filters Filter) []*models.Hit {
 }
 
 // ImportDashboards saves all dashboards matching query to configured location
-func (s *DashNGoImpl) ImportDashboards(filter Filter) []string {
+func (s *DashNGoImpl) ImportDashboards(filter filters.Filter) []string {
 	var (
 		boardLinks []*models.Hit
 		rawBoard   []byte
@@ -180,7 +256,7 @@ func (s *DashNGoImpl) createdFolder(folderName string) (int64, error) {
 
 // ExportDashboards finds all the dashboards in the configured location and exports them to grafana.
 // if the folder doesn't exist, it'll be created.
-func (s *DashNGoImpl) ExportDashboards(filters Filter) {
+func (s *DashNGoImpl) ExportDashboards(filterReq filters.Filter) {
 
 	var (
 		rawBoard   []byte
@@ -193,23 +269,19 @@ func (s *DashNGoImpl) ExportDashboards(filters Filter) {
 		log.WithError(err).Fatal("unable to find any files to export from storage engine")
 	}
 	//Delete all dashboards that match prior to import
-	s.DeleteAllDashboards(filters)
+	s.DeleteAllDashboards(filterReq)
 
-	folderMap := getFolderNameIDMap(s.ListFolder(nil))
+	folderMap := getFolderNameIDMap(s.ListFolder(NewFolderFilter()))
 
 	// Fallback on defaults
-	if filters == nil {
-		filters = &DashboardFilter{}
+	if filterReq == nil {
+		filterReq = NewDashboardFilter("", "", "")
 	}
-	validFolders := filters.GetFolders()
+	validFolders := filterReq.GetEntity(filters.FolderFilter)
 	for _, file := range filesInDir {
 		baseFile := filepath.Base(file)
 		baseFile = strings.ReplaceAll(baseFile, ".json", "")
-		validateMap := map[string]string{FolderFilter: folderName, DashFilter: baseFile}
-		//If folder OR slug is filtered, then skip if it doesn't match
-		if !filters.Validate(validateMap) {
-			continue
-		}
+
 		if !strings.HasSuffix(file, ".json") {
 			log.Warnf("Only json files are supported, skipping %s", file)
 			continue
@@ -237,14 +309,19 @@ func (s *DashNGoImpl) ExportDashboards(filters Filter) {
 			log.Debugf("Skipping file %s, doesn't match any valid folders", file)
 			continue
 		}
+		validateMap := map[filters.FilterType]string{filters.FolderFilter: folderName, filters.DashFilter: baseFile}
+		//If folder OR slug is filtered, then skip if it doesn't match
+		if !filterReq.ValidateAll(validateMap) {
+			continue
+		}
 
 		if folderName == DefaultFolderName {
 			folderId = DefaultFolderId
 		} else {
 			if val, ok := folderMap[folderName]; ok {
 				folderId = val
 			} else {
-				if filters.Validate(validateMap) {
+				if filterReq.ValidateAll(validateMap) {
 					id, folderErr := s.createdFolder(folderName)
 					if folderErr != nil {
 						log.Panic("Unable to create required folder")
@@ -278,12 +355,12 @@ func (s *DashNGoImpl) ExportDashboards(filters Filter) {
 
 // DeleteAllDashboards clears all current dashboards being monitored.  Any folder not white listed
 // will not be affected
-func (s *DashNGoImpl) DeleteAllDashboards(filter Filter) []string {
+func (s *DashNGoImpl) DeleteAllDashboards(filter filters.Filter) []string {
 	var dashboardListing = make([]string, 0)
 
 	items := s.ListDashboards(filter)
 	for _, item := range items {
-		if filter.Validate(map[string]string{FolderFilter: item.FolderTitle, DashFilter: item.Slug}) {
+		if filter.ValidateAll(map[filters.FilterType]string{filters.FolderFilter: item.FolderTitle, filters.DashFilter: item.Slug}) {
 			dp := dashboards.NewDeleteDashboardByUIDParams()
 			dp.UID = item.UID
 			_, err := s.client.Dashboards.DeleteDashboardByUID(dp, s.getAuth())