Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* allow non-email usernames * Cache the mediaconvert endpoint in order to avoid getting throttled on the DescribeEndpoints API. * allow input text to be empty * Add support for new languages in AWS Translate and Transcribe * Add support for new languages in AWS Translate and Transcribe * V0.1.6 bug fixes (#140) * allow non-email usernames * Cache the mediaconvert endpoint in order to avoid getting throttled on the DescribeEndpoints API. * allow input text to be empty * Add support for new languages in AWS Translate and Transcribe * Add support for new languages in AWS Translate and Transcribe * fix python 3.6 build errors and add support for python 3.8 * Fix markdown anchor for glossary * add support to delete an asset from elasticsearch (#142) * fix template validation error that happens when DeployAnalyticsPipeline=false but DeployDemoSite=true * Mitigate XSS threats (#147) * add subresource integrity (SRI) checksums so browsers can verify that the files they fetch are delivered without unexpected manipulation. * move runtime configs from .env to /public/runtimeConfig.json * webapp code cleanup * webapp code cleanup * Updated tests (#149) This PR focuses on scoping IAM policies with least privalege. Along the way we have also improved the organization of build scripts and unit tests so they're easier to use. Summary: * Least privalege concerns were achieved by updating Cloud Formation templates to resolve issues reported by cfn_nag and viperlight * We used to have many run_test.sh scripts to run unit tests. These have been consolidated into one script, tests/run_tests.sh, which you can run like this: `echo "$REGION \n $MIE_STACK_NAME \n $MIE_USERNAME \n $MIE_PASSWORD" | ./tests/run_tests.sh` Details: * a pass at refactoring iam roles/policies * refactor tests to use media in dataplane bucket, big test overhaul, small IAM changes for dataplane * do not assume the user has put the region at the end of the bucket name * Remove sam_translate from dataplaneapi and workflowapi. Organize the code and output so it's easier to follow. Access MIE Helper package from source/lib/ instead of /lib. * Apply bash syntax optimizations * Access MIE Helper package from source/lib/ instead of /lib. * update lib path to mie helper * remove redundant doc * add stream encryption to fix cfn_nag warning * remove sam-translate.py files * remove old /webapp and /lib * remove old /webapp and /lib * rename license file per AWS guidelines * rename notice file per AWS guidelines * output misc debug info * move tests/ into source/ Co-authored-by: Ian Downard <[email protected]> * Add mediainfo and transcode operators (#150) Resolved Issues: #32 #138 #152 #151 #128 #153 #154 #156 #157 Summary of changes: 1. added proxy encode to mediaconvert job that generates thumbnails 2. added MediaInfo libraries to MIE lambda layer. Also published these layers in the Technical 3. Marketing public S3 buckets. 4. added MediaInfo operator to MIE Complete Workflow and show mediainfo data in webui 5. major organization improvements in the build script 6. fixed minor webpack warnings 7. Added support for videos without spoken words 8. Added support for videos without any audio tracks 9. Added security measures to prevent users from uploading invalid media files Details: * Add mediainfo operator * Add MediaInfo library to MIE lambda layer * avoid webpack warnings about package size * fix compile-time jquery warning * remove unused requirements file * minor code cleanup * add log statement so we're consistent with other components * show mediainfo data in analysis page * explain how to enable hot-reload in dev mode * Explain how to validate data in elasticsearch. * Explain how to read/write metadata from one operator to another via workflow output objects. * skip comprehend operators when transcript is empty * skip comprehend operators when transcript is empty * skip transcribe if video is silent * use proxy encoded video for Rekognition operators * recognize more image file types when determining what to use for thumbnail * use a consistent print statement for logging the incoming lambda event object * Now that we're supporting media formats besides mp4 and jpg, use a generic image or video media type. We can't assume "video/mp4" or "image/jpg" anymore. * Remind developers that workflow attributes must be non-empty strings. * Add transcode to mediaconvert job. Use that for the proxy encode input to downstream operators. * Move transcribe operation from mediaconvert operator to thumbnail operator. The thumbnail operator now superseeds the old mediaconvert operator. We've disable old mediaconvert operator. After testing, we can remove the old mediaconvert operator. * Avoid drawing boxes outside the dimensions of the video player. * Thumbnail operator needs a check-status function now that it includes transcode. This commit adds that check-status function to the build script. * minor edit, just to reorder packages to improve readability * Move thumbnail operator to prelim stage so all mediaconvert outputs are ready before analysis operators begin. * avoid showing undefined mediainfo attributes * use free tier for elasticsearch domain * change header title to AWS Content Analysis * validate file types before upload * build layer for python 3.8 runtime * explain how to validate that the layer includes certain libraries * add PointInTimeRecoveryEnabled and HTTP (non-ssl) Deny rule to dataplane bucket * add versioning to S3 bucket * validate file type before upload and enable Mediainfo for image workflow * consolidate the code for checking image types * use webpack's default devServer https option * support all caps filenames * remove input media from upload/ after copying it to private/assets/[asset_id]/input/ * if input file is not a valid media file then remove it from S3 * Get mediaconvert endpoint from cache if available * Specify thumbnail as the first mediaconvert job so the thumbnail images become available as soon as possible. This lessens the likelihood of seeing broken thumbnail images in the webui. * Add Mediainfo to Image workflow and allow Mediainfo to delete files from S3. * minor edit to remove unnecessary whitespace * minor edit to fix a 'key not found' exception that occurred when testing an empty workflow execution request (e.g. POST {} payload to /api/workflow/execution) * Add Mediainfo to image workflow * minor remove errand comma * add CloudFormation string functions so we can use (lower case) stack name for mie website bucket * fix bug in error messages for invalid file types * fix yaml syntax errors * fix invalid table query when invoking a GET on $WORKFLOW_API_ENDPOINT/workflow/execution/status/Error * fix "key not found" error that occurs running workflows that include transcribe but not mediainfo * 1) Update workflow configs and 2) upload media prior to every workflow execution because dataplane now deletes the uploaded media after copying it to private/assets/. * upload media prior to workflow execution because dataplane now deletes the uploaded media after copying it to private/assets/. * 1) Update workflow configs and 2) upload media prior to every workflow execution because dataplane now deletes the uploaded media after copying it to private/assets/. * cleanup comments * Use app.current_request.raw_body.decode instead of app.current_request.json_body in order to work around a bug in Chalice whereby it returns None for json_body. Reference: https://stackoverflow.com/questions/52789943/cannot-access-the-request-json-body-when-using-chalice * append a unique id to image files uploaded to s3 so there are no conflicts between multiple threads running this concurrency test * Handle the HTTP 409 and 500 errors that happen when tests don't clean up properly. * add cost information * minor edits * minor edits * minor edits * minor edits * fix bug detection silent videos * bump up the python version * bump up the python version * Rek detect text in video support (#158) * rek text detection functionality * bug fixes for player markers and readdition of accidentally deleted code for text detection * fix string operation to determine file type * get input video from ProxyEncode (#168) * get input video from ProxyEncode * add new region support for Rekognition (#163) * allow users to upload videos with formats supported by mediaconvert (#169) * get input video from ProxyEncode * add new region support for Rekognition * allow users to upload videos with formats supported by mediaconvert (#164) * allow users to upload videos with formats supported by mediaconvert * Allow users to upload webm files. * fix bug with determining key to proxy encode mp4 * fix bug with determining key to proxy encode mp4 (#170) * get input video from ProxyEncode * add new region support for Rekognition * allow users to upload videos with formats supported by mediaconvert * Allow users to upload webm files. * fix bug with determining key to proxy encode mp4 * Disable versioning on dataplane bucket (#171) * Disable versioning on dataplane bucket because so that bucket can be removed more easily * minor edit * support for rerunning analysis on an existing asset (#175) * support for rerunning analysis on an existing asset * bug fix in webapp code * fix formatting issues after merge and update status to be polled by wf id * Add gitter chat info (#182) * Bumps [jquery](https://github.com/jquery/jquery) from 1.12.4 to 3.4.1. - [Release notes](https://github.com/jquery/jquery/releases) - [Commits](jquery/[email protected]) * add gitter channel info * Bumps [jquery](https://github.com/jquery/jquery) from 1.12.4 to 3.4.1. (#181) - [Release notes](https://github.com/jquery/jquery/releases) - [Commits](jquery/[email protected]) * Fix mediainfo (#180) * remove VersioningConfiguration on S3 bucket since that makes it much harder for AWS account owners to delete the bucket. * MediaInfo version 19.09 works but 20.03 does not. Use to 19.09 instead of latest. * update one-click deploy links for release version 0.1.7 * testing buildspec * version bump python version in buildspec * remove unneeded quotes from build command * Change distribution bucket instructions (#189) Previously, the instruction was to created a distribution bucket named $DIST_OUTPUT_BUCKET-$REGION, but now in `deployment/build-s3-dist.sh` it's expected to be just $DIST_OUTPUT_BUCKET. * Init of build pipeline (#193) * working build pipeline * fix testing spec filename * persist build user * Add logo (#194) The clapperboard, representing *multimedia*, is centered inside a crosshair, representing *under extreme scrutiny*. This symbol is available from [nounproject](https://thenounproject.com/icon/1815092/). The font is Engineering Plot, https://www.dafont.com/engineering-plot.font which conveys the scaffolding nature of MIE. * Move my forked version of the isolated MIE backend into the main repo for collab (#196) * init isolated mie framework * removed unneeded email param * add restapi ids to outputs * add todo for cors * Update README.md (#197) Improve instructions in the README: * fix references to old MediaInsightsEngine repository name * use docker port forwarding to enable developers to see the result of npm run serve on their local machine * Update media-insights-stack.yaml (#198) fix PolicyName typo * Prevent duplication of this.entities (#201) If user switches from Entities to KeyPhrases tab and back, this.entities doubles in size. To prevent this, we can employ the same method of clearing memory that is used in ComprehendKeyPhrases * Avoid linking to step functions for queued workflows because that link will break since the step function doesn't exist yet. (#210) * Added Cognito Identity Pool ID to the output of CF (#211) Add IDENTITY_POOL_ID to stack outputs in order to make it easier for users to find the values they will need for the `webapp/public/runtimeConfig.json` file when trying to run the webapp locally on their laptop. * change logo. The MIE team agreed to use the 3d black and white logo w… (#200) * change logo. The MIE team agreed to use the 3d black and white logo without a slogan. * move logo files to doc/images * Update gui readme (#202) * add instructions for creating new accounts for the GUI and remove out-of-date instructions for running the webapp. * Add quantitative cost info * fix type-o * add cursor usage info * document limitations * update 3rd party licenses to include every package listed in package.json * remove local dist and package files after build * remove license file form MIE lambda helper. This was left over from when the lambda helper used to be in its own repo * Remove reference to old MediaInsightsEngineLambdaHelper repo. It used to be managed in a different repo but now it's part of this repo. * Video segment detection / v0.1.8 one click links (#215) * working segment detection v1 * working segment detection w/ api changes * added end scene pause functionality and pagination to scene detection tables * fix webapp deploy bug * reformat readme for simpler installation * updated readme with instructions for installtion * remove values from runtimeConfig and set sriplugin to true * Added .vscode to gitignore * Updated MIE_ACCESS_TOKEN to retreive token from correct path in the export statement under IMPLEMENTATION_GUIDE. Updated MieDataplaneApiHandlerRolePolicy to include ListBucket on the Dataplane bucket to ensure S3 NoSuchKey Error message is given instead of AccessDenied when accessing a missing S3 key. * Added ListBucket to Dataplane bucket policy for better debugging and minor documentation correction (#235) Updated the Dataplane API Handler's Role policy to include a ListBucket action on the Dataplane S3 bucket. This is done so that the developer gets a NoSuchKey error when accessing an invalid S3 key instead of getting AccessDenied. The incorrect message makes it hard to debug especially when all required permissions for execution of the Lambda exist. Updated the path under Implementation guide to reflect the correct path when exporting the MIE_ACCESS_TOKEN. Currently: $MIE_DEVELOPMENT_HOME/tests/getAccessToken.py. Proposed change: $MIE_DEVELOPMENT_HOME/source/tests/getAccessToken.py Added .vscode/ to gitignore as a QOL improvement for VSCode users. * more updates to backend mie stack * add addl exports to template * fix #236 * adjust template url location * remove missed merge conflict text from stack template * remove commented-out code * use virtual-hosted style s3 paths * use virtual-hosted style s3 paths * Update documentation to reflect the new MIE backend * fix s3 copy error * Update installation instructions. Move implementation guide to Media Insights front-end repo. * adjust images * adjust images * minor update * minor update * minor update * minor update * minor update * remove changes to gitignore and IMPLEMENTATION_GUIDE.md * Remove a modified file from pull request * Remove a modified file from pull request * Remove a modified file from pull request * fix #206 (#246) * minor update * minor update * change rodeolabz s3 folder * change rodeolabz s3 folder * minor update * change rodeolabz s3 folder * change rodeolabz s3 folder * add postman screenshot * update version used in the one-click deploy links Co-authored-by: brand161 <[email protected]> Co-authored-by: Brandon Dold <[email protected]> Co-authored-by: brandold <[email protected]> Co-authored-by: Tulio Casagrande <[email protected]> Co-authored-by: Anton <[email protected]> Co-authored-by: Rajesh <[email protected]>
- Loading branch information
7 people
authored
Oct 7, 2020
1 parent
a663d95
commit 49df487