Skip to content

Commit 30fc121

Browse files
igfasouzaigfasouza
committed
add new samples for service connector hub, stream and Functions integrations
1 parent 56583d3 commit 30fc121

File tree

13 files changed

+511
-0
lines changed

13 files changed

+511
-0
lines changed

‎THIRD_PARTY_LICENSES.txt

+171
Original file line numberDiff line numberDiff line change
@@ -400,6 +400,177 @@ See the License for the specific language governing permissions and
400400
limitations under the License.
401401

402402

403+
--------------------------------------------------------------------------------
404+
405+
Pandas
406+
407+
BSD 3-Clause License
408+
409+
Copyright (c) 2008-2012, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team
410+
All rights reserved.
411+
412+
Redistribution and use in source and binary forms, with or without
413+
modification, are permitted provided that the following conditions are met:
414+
415+
* Redistributions of source code must retain the above copyright notice, this
416+
list of conditions and the following disclaimer.
417+
418+
* Redistributions in binary form must reproduce the above copyright notice,
419+
this list of conditions and the following disclaimer in the documentation
420+
and/or other materials provided with the distribution.
421+
422+
* Neither the name of the copyright holder nor the names of its
423+
contributors may be used to endorse or promote products derived from
424+
this software without specific prior written permission.
425+
426+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
427+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
428+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
429+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
430+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
431+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
432+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
433+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
434+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
435+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
436+
--------------
437+
Fourth party dependencies
438+
----------------
439+
pandas is dependent on the following :
440+
441+
Python Package : numpy - Version used for numpy is 1.16.2 [There is a new LT also created for the same]
442+
Version 1.16.2
443+
Maintainer/Licensor: NumPy Developers
444+
License: BSD License
445+
Page : https://pypi.org/project/numpy/
446+
447+
Copyright (c) 2005-2019, NumPy Developers.
448+
All rights reserved.
449+
450+
Redistribution and use in source and binary forms, with or without
451+
modification, are permitted provided that the following conditions are
452+
met:
453+
454+
* Redistributions of source code must retain the above copyright
455+
notice, this list of conditions and the following disclaimer.
456+
457+
* Redistributions in binary form must reproduce the above
458+
copyright notice, this list of conditions and the following
459+
disclaimer in the documentation and/or other materials provided
460+
with the distribution.
461+
462+
* Neither the name of the NumPy Developers nor the names of any
463+
contributors may be used to endorse or promote products derived
464+
from this software without specific prior written permission.
465+
466+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
467+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
468+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
469+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
470+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
471+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
472+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
473+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
474+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
475+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
476+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
477+
478+
479+
Python Package : python-dateutil
480+
Version 2.8.0
481+
Author: Gustavo Niemeyer
482+
Maintainer: Paul Ganssle
483+
License: Apache Software License,
484+
BSD License (Dual License)
485+
Page : https://pypi.org/project/python-dateutil/
486+
dateutil - Extensions to the standard Python datetime module.
487+
488+
Copyright (c) 2003-2011 - Gustavo Niemeyer <[email protected]>
489+
Copyright (c) 2012 - Tomi Pieviläinen <[email protected]>
490+
491+
All rights reserved.
492+
493+
Redistribution and use in source and binary forms, with or without
494+
modification, are permitted provided that the following conditions are met:
495+
496+
* Redistributions of source code must retain the above copyright notice,
497+
this list of conditions and the following disclaimer.
498+
* Redistributions in binary form must reproduce the above copyright notice,
499+
this list of conditions and the following disclaimer in the documentation
500+
and/or other materials provided with the distribution.
501+
* Neither the name of the copyright holder nor the names of its
502+
contributors may be used to endorse or promote products derived from
503+
this software without specific prior written permission.
504+
505+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
506+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
507+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
508+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
509+
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
510+
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
511+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
512+
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
513+
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
514+
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
515+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
516+
517+
518+
519+
Python Package : six
520+
Version 1.12.0
521+
Author: Benjamin Peterson
522+
License: MIT License (MIT)
523+
Page : https://pypi.org/project/six/
524+
Copyright (c) 2010-2018 Benjamin Peterson
525+
526+
Permission is hereby granted, free of charge, to any person obtaining a copy of
527+
this software and associated documentation files (the "Software"), to deal in
528+
the Software without restriction, including without limitation the rights to
529+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
530+
the Software, and to permit persons to whom the Software is furnished to do so,
531+
subject to the following conditions:
532+
533+
The above copyright notice and this permission notice shall be included in all
534+
copies or substantial portions of the Software.
535+
536+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
537+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
538+
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
539+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
540+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
541+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
542+
543+
Python Package : pytz
544+
Version 2018.9
545+
Author: Stuart Bishop
546+
License: MIT License (MIT)
547+
Page : https://pypi.org/project/pytz//
548+
549+
Copyright (c) 2003-2005 Stuart Bishop <[email protected]>
550+
551+
Permission is hereby granted, free of charge, to any person obtaining a
552+
copy of this software and associated documentation files (the "Software"),
553+
to deal in the Software without restriction, including without limitation
554+
the rights to use, copy, modify, merge, publish, distribute, sublicense,
555+
and/or sell copies of the Software, and to permit persons to whom the
556+
Software is furnished to do so, subject to the following conditions:
557+
558+
The above copyright notice and this permission notice shall be included in
559+
all copies or substantial portions of the Software.
560+
561+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
562+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
563+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
564+
THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
565+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
566+
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
567+
DEALINGS IN THE SOFTWARE.
568+
569+
--------------------------------------------------------------------------------
570+
571+
pyarrow
572+
License: Apache License 2.0, a copy of the license is included at the end of this file.
573+
403574
================================================================================
404575
Apache License
405576
Version 2.0, January 2004

‎samples/oci-serviceconnector-streaming-json-to-csv-python/README.md

+84
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
## Convert JSON to CSV
2+
3+
This is a sample to convert JSON to CSV using Service Connector Hub Functions as a Task.
4+
5+
As you make your way through this tutorial, look out for this icon ![user input icon](./images/userinput.png).
6+
Whenever you see it, it's time for you to perform an action.
7+
8+
9+
## Prerequisites
10+
11+
Before you deploy this sample function, make sure you have run steps A, B
12+
and C of the [Oracle Functions Quick Start Guide for Cloud Shell](https://www.oracle.com/webfolder/technetwork/tutorials/infographics/oci_functions_cloudshell_quickview/functions_quickview_top/functions_quickview/index.html)
13+
* A - Set up your tenancy
14+
* B - Create application
15+
* C - Set up your Cloud Shell dev environment
16+
17+
18+
## List Applications
19+
20+
Assuming you have successfully completed the prerequisites, you should see your
21+
application in the list of applications.
22+
23+
```
24+
fn ls apps
25+
```
26+
27+
28+
## Review and customize the function
29+
30+
Review the following files in the current folder:
31+
* the code of the function, [func.py](./func.py)
32+
* its dependencies, [requirements.txt](./requirements.txt)
33+
* the function metadata, [func.yaml](./func.yaml)
34+
35+
36+
## Deploy the function
37+
38+
In Cloud Shell, run the `fn deploy` command to build *this* function and its dependencies as a Docker image,
39+
push the image to the specified Docker registry, and deploy *this* function to Oracle Functions
40+
in the application created earlier:
41+
42+
![user input icon](./images/userinput.png)
43+
```
44+
fn -v deploy --app <app-name>
45+
```
46+
e.g.,
47+
```
48+
fn -v deploy --app myapp
49+
```
50+
51+
### Test
52+
53+
In Cloud Shell, run the `fn invoke` ccommand to unit test this function:
54+
55+
![user input icon](./images/userinput.png)
56+
```
57+
fn invoke <app-name> <function-name> < test.json
58+
```
59+
e.g.,
60+
```
61+
fn invoke myapp oci-sch-stream-json-to-csv-python < test.json
62+
```
63+
64+
You should see the the converted CSV output returned by the function.
65+
66+
67+
## Create a Service Connector for reading from Stream 1, convert JSON to CSV and send to Stream 2
68+
69+
1. From the navigation menu, select **Logging**, and then select **Service Connectors**.
70+
71+
2. Click Create Connector, add a Name, Description, select the compartment, select the Source as **Streaming** and Target as **Streaming** and **Functions** as a Task.
72+
73+
3. On Configure Source connection, select the compartment, select the your Stream 1.
74+
75+
4. On Configure Target connection, select the compartment, select the your Stream 2. If prompted to create a policy for writing to Streaming, click Create.
76+
77+
5. On Configure Task connection, select the compartment and select the Function. If prompted to create a policy for writing to Functions, click Create.
78+
79+
80+
## Monitoring Functions and Service Connector
81+
82+
Make sure you configure basic observability for your function and connector using metrics, alarms and email alerts:
83+
* [Basic Guidance for Monitoring your Functions](../basic-observability/functions.md)
84+
* [Basic Guidance for Monitoring your Service Connector](../basic-observability/service-connector-hub.md)

‎samples/oci-serviceconnector-streaming-json-to-csv-python/func.py

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#
2+
# oci-serviceconnector-streaming-json-to-csv-python version 1.0.
3+
#
4+
# Copyright (c) 2021 Oracle, Inc. All rights reserved.
5+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
6+
#
7+
8+
import io
9+
import oci
10+
import json
11+
import requests
12+
import logging
13+
import base64
14+
import pandas as pd
15+
from fdk import response
16+
17+
18+
"""
19+
This Function converts JSON to CSV format
20+
"""
21+
def handler(ctx, data: io.BytesIO=None):
22+
logger = logging.getLogger()
23+
24+
try:
25+
logs = json.loads(data.getvalue())
26+
logger.info('Received {} entries.'.format(len(logs)))
27+
28+
for item in logs:
29+
if 'value' in item:
30+
item['value'] = base64_decode(item['value'])
31+
32+
if 'key' in item:
33+
item['key'] = base64_decode(item['key'])
34+
35+
df = pd.json_normalize(logs)
36+
csv_result = df.to_csv(index=False)
37+
return response.Response(ctx, status_code=200, response_data=csv_result, headers={"Content-Type": "text/csv"})
38+
39+
except (Exception, ValueError) as e:
40+
logger.error(str(e))
41+
raise
42+
43+
44+
def base64_decode(encoded):
45+
print(type(encoded))
46+
base64_bytes = encoded.encode('utf-8')
47+
message_bytes = base64.b64decode(base64_bytes)
48+
return message_bytes.decode('utf-8')

‎samples/oci-serviceconnector-streaming-json-to-csv-python/func.yaml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
schema_version: 20180708
2+
name: oci-sch-stream-json-to-csv-python
3+
version: 0.0.48
4+
runtime: python
5+
entrypoint: /python/bin/fdk /function/func.py handler
6+
memory: 256
7+
timeout: 300

‎samples/oci-serviceconnector-streaming-json-to-csv-python/images/userinput.png

2.96 KB
Loading

‎samples/oci-serviceconnector-streaming-json-to-csv-python/requirements.txt

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
fdk
2+
requests
3+
oci
4+
pandas
5+
pyarrow

‎samples/oci-serviceconnector-streaming-json-to-csv-python/test.json

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
[{"stream": "mynewstream","partition": "0","key": null,"value": "eyJuYW1lIjoiaWdvciIsICJsb2NhdGlvbiI6ICJEdWJsaW4ifQ==","offset": 0,"timestamp": "2018-04-26T01:03:06.051Z"},{"stream": "mynewstream","partition": "0","key": null,"value": "eyJuYW1lIjoiYnJpYW4iLCJmaWVsZDEiOiJ0ZXN0In0=","offset": 0,"timestamp": "2018-04-26T01:03:06.051Z"}]

0 commit comments

Comments
 (0)