diff --git a/HyperAuth/README.md b/HyperAuth/README.md index 126b9953..721fa5aa 100644 --- a/HyperAuth/README.md +++ b/HyperAuth/README.md @@ -87,15 +87,30 @@ LoadBalancer, NodePort type의 service 생성 가능 ## Step 2. SSL 인증서 생성 * 목적 : `HTTPS 인증을 위한 openssl 인증서를 생성하고 secret으로 변환` -* 생성 순서 : 아래 명령어를 실행하여 인증서 생성 및 secret을 생성 (Master Node의 특정 directory 내부에서 실행 권장) +* 생성 순서 : 아래 명령어를 실행하여 인증서 생성 및 secret을 생성 (Master Node의 특정 directory 내부에서 실행 권장) (인증서 기한 10년 2020-12-04 이후부터 적용, 이전은 1년) ```bash - $ openssl req -newkey rsa:4096 -nodes -sha256 -keyout hyperauth.key -x509 -subj "/C=KR/ST=Seoul/O=tmax/CN=$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)" -days 365 -config <(cat /etc/ssl/openssl.cnf <(printf "[v3_ca]\nsubjectAltName=IP:$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)")) -out hyperauth.crt + $ openssl req -newkey rsa:4096 -nodes -sha256 -keyout hyperauth.key -x509 -subj "/C=KR/ST=Seoul/O=tmax/CN=$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)" -days 3650 -config <(cat /etc/ssl/openssl.cnf <(printf "[v3_ca]\nsubjectAltName=IP:$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)")) -out hyperauth.crt + $ CentOS의 경우 : openssl req -newkey rsa:4096 -nodes -sha256 -keyout hyperauth.key -x509 -subj "/C=KR/ST=Seoul/O=tmax/CN=$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)" -days 3650 -config <(cat /etc/pki/tls/openssl.cnf <(printf "[v3_ca]\nsubjectAltName=IP:$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)")) -out hyperauth.crt $ kubectl create secret tls hyperauth-https-secret --cert=./hyperauth.crt --key=./hyperauth.key -n hyperauth $ cp hyperauth.crt /etc/kubernetes/pki/hyperauth.crt ``` * 비고 : * Kubernetes Master가 다중화 된 경우, hyperauth.crt를 각 Master 노드들의 /etc/kubernetes/pki/hyperauth.crt 로 cp - +* 인증서 만료 됐을때 + * 인증서 만료 확인 : openssl x509 -in hyperauth.crt -noout -dates + * 인증서 재발급 및 secret 생성 적용 +```bash + // 10년 짜리 인증서 재발급 + $ openssl req -newkey rsa:4096 -nodes -sha256 -keyout hyperauth.key -x509 -subj "/C=KR/ST=Seoul/O=tmax/CN=$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)" -days 3650 -config <(cat /etc/ssl/openssl.cnf <(printf "[v3_ca]\nsubjectAltName=IP:$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)")) -out hyperauth.crt + $ CentOS의 경우 : openssl req -newkey rsa:4096 -nodes -sha256 -keyout hyperauth.key -x509 -subj "/C=KR/ST=Seoul/O=tmax/CN=(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)" -days 3650 -config <(cat /etc/pki/tls/openssl.cnf <(printf "[v3_ca]\nsubjectAltName=IP:$(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7)")) -out hyperauth.crt + + // hyperauth-https-secret-renewed 라는 이름으로 secret을 새롭게 만든다. + $ kubectl create secret tls hyperauth-https-secret-renewed --cert=./hyperauth.crt --key=./hyperauth.key -n hyperauth + $ cp hyperauth.crt /etc/kubernetes/pki/hyperauth.crt + + // hyperauth deploy의 mount secret 이름을 바꾼다. + $ kubectl patch deployment hyperauth -n hyperauth --patch '{"spec":{"template":{"spec":{"volumes":[{"name":"ssl","secret":{"secretName":"hyperauth-https-secret-renewed"}}]}}}}' +``` ## Step 3. HyperAuth Deployment 배포 * 목적 : `HyperAuth 설치` @@ -104,18 +119,23 @@ LoadBalancer, NodePort type의 service 생성 가능 * HyperAuth Admin Console에 접속 * `kubectl get svc hyperauth -n hyperauth` 명령어로 IP 확인 * 계정 : admin/admin - * Manage > Users > Add user 에서 admin-tmax.co.kr 계정을 생성 + * Master > Add realm > Import - Select file 에 [3.tmax_realm_export.json](manifest/3.tmax_realm_export.json) 을 추가하여 Realm Import + * tmaxRealm > Manage > Users > Add user 에서 admin-tmax.co.kr 계정을 생성 * Name : admin-tmax.co.kr * Email : 관리자 전용 email * Manage > Users > admin-tmax.co.kr UserDetail > Credentials 에서 password 재설정 - * Master > Add realm > Import - Select file 에 [3.tmax_realm_export.json](manifest/3.tmax_realm_export.json) 을 추가하여 Realm Import - * 위의 HyperAuth Admin Console 접속 부터 tmax realm import까지의 과정을 tmaxRealmImport.sh로 대신할 수 있음 * {HYPERAUTH_SERVICE_IP} = $(kubectl describe service hyperauth -n hyperauth | grep 'LoadBalancer Ingress' | cut -d ' ' -f7) * {HYPERCLOUD-CONSOLE_IP} = $(kubectl describe service console-lb -n console-system | grep 'LoadBalancer Ingress' | cut -d ' ' -f7) * 실행 : ./tmaxRealmImport.sh {HYPERAUTH_SERVICE_IP} {HYPERCLOUD-CONSOLE_IP} - + * 메일 정보 입력 + * Realm Settings - Email에 필수 입력 란을 채워야 메일 전송이 이루어진다. + * Host, From 을 입력한다. ex) mail.tmax.co.kr, tmaxcloud_ck@tmax.co.kr + * Enable Authentication을 ON 으로 바꾼다. + * Username, Password를 입력한다. (mail.tmax.co.kr에 로그인 할 수 있는 관리자 계정) + * Manage account - Account 의 Username, Email, First name, Last name을 입력한다. + ## Step 4. Kubernetes OIDC 연동 * 목적 : `Kubernetes의 RBAC 시스템과 HyperAuth 인증 연동` * 생성 순서 : @@ -133,4 +153,20 @@ LoadBalancer, NodePort type의 service 생성 가능 ``` * 비고 : - * 자동으로 kube-apiserver 가 재기동 됨 + * 자동으로 kube-apiserver 가 재기동 됨 + +#HyperAuth 유저 Migration Guide +## Hypercloud 4.0 User (CRD) Hyperauth 로 Migration +* 순서 : +1. https://github.com/tmax-cloud/hypercloud-operator/blob/master/scripts/userMigrationFromK8s.sh 파일을 migration하려는 환경에 위치시킨다. +2. chmod +777 userMigrationFromK8s.sh +3. kubectl get svc -n hyperauth + * hyperauth 서비스의 http ip:port 를 복사한다. ex) 192.168.6.223:8080 +4. ./userMigrationFromK8s.sh 192.168.6.223:8080 으로 call 한다. + * 전제조건 : tmax realm은 만들어진 상태라고 가정한다. + * 이미 존재해서 중복되는 아이디가 있을 경우, 해당 유저만 migration이 되지 않는다. +5. hyperauth admin console에서 tmax realm의 user가 추가된 것을 확인한다, attribute도 추가된 것을 확인한다. ( phone, department, dateOfBirth, description 지원 ) +6. 비밀번호 Setting 하기 + * client 로그인 화면에서 Forgot Password? 를 클릭한다. + * 본인이 기존에 쓰던 userId를 Username에 입력한다. + * 등록되어 있던 메일로 전송 된 비밀번호 재설정 링크를 통해 비밀번호를 설정한다. diff --git a/HyperAuth/manifest/3.tmax-realm-export.json b/HyperAuth/manifest/3.tmax-realm-export.json index 929e62c4..d124f15f 100644 --- a/HyperAuth/manifest/3.tmax-realm-export.json +++ b/HyperAuth/manifest/3.tmax-realm-export.json @@ -2,12 +2,12 @@ "id": "tmax", "realm": "tmax", "displayName": "tmax", - "notBefore": 1597835890, + "notBefore": 1602720991, "revokeRefreshToken": false, "refreshTokenMaxReuse": 0, "accessTokenLifespan": 300, "accessTokenLifespanForImplicitFlow": 900, - "ssoSessionIdleTimeout": 3600, + "ssoSessionIdleTimeout": 1800, "ssoSessionMaxLifespan": 36000, "ssoSessionIdleTimeoutRememberMe": 0, "ssoSessionMaxLifespanRememberMe": 0, @@ -24,23 +24,40 @@ "enabled": true, "sslRequired": "external", "registrationAllowed": true, - "registrationEmailAsUsername": false, + "registrationEmailAsUsername": true, "rememberMe": true, - "verifyEmail": false, - "loginWithEmailAllowed": false, - "duplicateEmailsAllowed": true, + "verifyEmail": true, + "loginWithEmailAllowed": true, + "duplicateEmailsAllowed": false, "resetPasswordAllowed": true, "editUsernameAllowed": false, - "bruteForceProtected": false, + "bruteForceProtected": true, "permanentLockout": false, "maxFailureWaitSeconds": 900, "minimumQuickLoginWaitSeconds": 60, "waitIncrementSeconds": 60, - "quickLoginCheckMilliSeconds": 1000, - "maxDeltaTimeSeconds": 43200, - "failureFactor": 30, + "quickLoginCheckMilliSeconds": 500, + "maxDeltaTimeSeconds": 86400, + "failureFactor": 10, "roles": { "realm": [ + { + "id": "f9b355c3-6676-4fc8-9947-374d8773ffce", + "name": "teespace middle admin", + "composite": true, + "composites": { + "client": { + "realm-management": [ + "query-groups", + "view-users", + "query-users" + ] + } + }, + "clientRole": false, + "containerId": "tmax", + "attributes": {} + }, { "id": "d449e9d4-b460-4ff4-b557-265f02683320", "name": "uma_authorization", @@ -50,6 +67,14 @@ "containerId": "tmax", "attributes": {} }, + { + "id": "21b13295-61d1-436a-9ccf-9ec1c0bfdd3f", + "name": "test", + "composite": false, + "clientRole": false, + "containerId": "tmax", + "attributes": {} + }, { "id": "4d951274-8c75-42d0-add0-832133fc574f", "name": "tmax realm user, group all role", @@ -141,6 +166,17 @@ } ], "client": { + "gatekeeper": [ + { + "id": "74fd8dc5-c7ba-48cf-a38c-e09bf9d1509e", + "name": "uma_protection", + "composite": false, + "clientRole": true, + "containerId": "fe8716c3-728c-4d78-9a91-cc31176a01bf", + "attributes": {} + } + ], + "console": [], "realm-management": [ { "id": "38840ce8-dced-4916-bf86-c4c3b7e84c20", @@ -356,7 +392,16 @@ "kiali": [], "security-admin-console": [], "admin-cli": [], - "hypercloud4": [], + "hypercloud4": [ + { + "id": "f243fe37-24ad-4a5c-b42a-e3e7c67a8baf", + "name": "uma_protection", + "composite": false, + "clientRole": true, + "containerId": "8e0f17ee-fd09-4da2-9b4f-270e49dc0e6c", + "attributes": {} + } + ], "notebook-gatekeeper": [], "account-console": [], "broker": [ @@ -439,7 +484,8 @@ "containerId": "b48941f1-5c5e-4eae-854c-6057385d8772", "attributes": {} } - ] + ], + "tmaxcloud": [] } }, "groups": [ @@ -465,6 +511,15 @@ "clientRoles": {}, "subGroups": [] }, + { + "id": "20be74a8-e8fd-43df-bb35-c3b3c42ad91d", + "name": "nodejs", + "path": "/nodejs", + "attributes": {}, + "realmRoles": [], + "clientRoles": {}, + "subGroups": [] + }, { "id": "fd8e1606-6daf-46a7-84b0-2e5e2320fcbf", "name": "tmaxcloud", @@ -482,7 +537,7 @@ "requiredCredentials": [ "password" ], - "passwordPolicy": "notUsername(undefined) and length(5)", + "passwordPolicy": "specialChars(1) and upperCase(1) and lowerCase(1) and length(9) and digits(1)", "otpPolicyType": "totp", "otpPolicyAlgorithm": "HmacSHA1", "otpPolicyInitialCounter": 0, @@ -517,6 +572,34 @@ "webAuthnPolicyPasswordlessCreateTimeout": 0, "webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": false, "webAuthnPolicyPasswordlessAcceptableAaguids": [], + "users": [ + { + "id": "a1618fd7-00c1-4504-9b7f-85195f04d762", + "createdTimestamp": 1600060522980, + "username": "service-account-gatekeeper", + "enabled": true, + "totp": false, + "emailVerified": false, + "serviceAccountClientId": "gatekeeper", + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "uma_authorization", + "offline_access" + ], + "clientRoles": { + "gatekeeper": [ + "uma_protection" + ], + "account": [ + "view-profile", + "manage-account" + ] + }, + "notBefore": 0, + "groups": [] + } + ], "scopeMappings": [ { "clientScope": "offline_access", @@ -763,6 +846,171 @@ "microprofile-jwt" ] }, + { + "id": "3542dc09-b2be-4ec6-bd95-6c95516ed6df", + "clientId": "console", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "secret": "**********", + "redirectUris": [ + "*" + ], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": false, + "publicClient": false, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "saml.assertion.signature": "false", + "saml.multivalued.roles": "false", + "saml.force.post.binding": "false", + "saml.encrypt": "false", + "saml.server.signature": "false", + "saml.server.signature.keyinfo.ext": "false", + "exclude.session.state.from.auth.response": "false", + "saml_force_name_id_format": "false", + "saml.client.signature": "false", + "tls.client.certificate.bound.access.tokens": "false", + "saml.authnstatement": "false", + "display.on.consent.screen": "false", + "saml.onetimeuse.condition": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "defaultClientScopes": [ + "web-origins", + "role_list", + "profile", + "roles", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "fe8716c3-728c-4d78-9a91-cc31176a01bf", + "clientId": "gatekeeper", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "secret": "**********", + "redirectUris": [ + "*" + ], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": true, + "publicClient": false, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "saml.assertion.signature": "false", + "access.token.lifespan": "259200", + "saml.multivalued.roles": "false", + "saml.force.post.binding": "false", + "saml.encrypt": "false", + "saml.server.signature": "false", + "saml.server.signature.keyinfo.ext": "false", + "exclude.session.state.from.auth.response": "false", + "saml_force_name_id_format": "false", + "saml.client.signature": "false", + "tls.client.certificate.bound.access.tokens": "false", + "saml.authnstatement": "false", + "display.on.consent.screen": "false", + "saml.onetimeuse.condition": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "protocolMappers": [ + { + "id": "c142e9e1-d54c-4e37-91d1-126338475149", + "name": "Client Host", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientHost", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientHost", + "jsonType.label": "String" + } + }, + { + "id": "fc566909-329c-436d-a290-596cddf92383", + "name": "Client ID", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientId", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientId", + "jsonType.label": "String" + } + }, + { + "id": "41194a05-26f3-402a-8424-c2022493fd52", + "name": "test", + "protocol": "openid-connect", + "protocolMapper": "oidc-role-name-mapper", + "consentRequired": false, + "config": { + "role": "test", + "new.role.name": "test" + } + }, + { + "id": "2016e5b4-c63c-4997-8827-89152028b3eb", + "name": "Client IP Address", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientAddress", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientAddress", + "jsonType.label": "String" + } + } + ], + "defaultClientScopes": [ + "web-origins", + "gatekeeper", + "role_list", + "profile", + "roles", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "offline_access", + "microprofile-jwt" + ] + }, { "id": "8e0f17ee-fd09-4da2-9b4f-270e49dc0e6c", "clientId": "hypercloud4", @@ -778,11 +1026,12 @@ "secret": "**********", "redirectUris": [ "https://172.22.6.14/*", - "https://17.17.17.*", + "https://172.22.6.14/kiali/*", "https://192.168.6.125:9000/*", - "*", + "http://192.168.8.*", "https://192.168.8.*", "https://172.22.1.12/*", + "http://192.168.8.25/complete.html", "https://172.22.6.2:31304/*" ], "webOrigins": [ @@ -792,7 +1041,7 @@ "bearerOnly": false, "consentRequired": false, "standardFlowEnabled": true, - "implicitFlowEnabled": false, + "implicitFlowEnabled": true, "directAccessGrantsEnabled": true, "serviceAccountsEnabled": false, "publicClient": true, @@ -820,6 +1069,34 @@ "fullScopeAllowed": true, "nodeReRegistrationTimeout": -1, "protocolMappers": [ + { + "id": "bf6c53a4-d685-4d3f-a598-9911a85c603e", + "name": "Client ID", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientId", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientId", + "jsonType.label": "String" + } + }, + { + "id": "c17dad3c-0400-49a7-9530-a63ba38491ad", + "name": "Client IP Address", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientAddress", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientAddress", + "jsonType.label": "String" + } + }, { "id": "999a7393-6ce5-4e80-8123-f28f8d5198f3", "name": "phone", @@ -835,6 +1112,20 @@ "jsonType.label": "String" } }, + { + "id": "0c9d3439-ef66-4b9f-95af-26eaad41ebc7", + "name": "Client Host", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientHost", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientHost", + "jsonType.label": "String" + } + }, { "id": "eb050e3a-dd58-4d3d-9a8e-6d90537f6759", "name": "banlist", @@ -871,19 +1162,19 @@ "clientId": "kiali", "name": "kiali", "description": "kiali", - "rootUrl": "http://172.22.6.3:32587/kiali/", - "adminUrl": "http://172.22.6.3:32587/kiali/", - "baseUrl": "/", + "rootUrl": "http://172.22.6.3:32588/kiali/", + "adminUrl": "http://172.22.6.3:32588/kiali/", + "baseUrl": "", "surrogateAuthRequired": false, "enabled": true, "alwaysDisplayInConsole": false, "clientAuthenticatorType": "client-secret", "secret": "**********", "redirectUris": [ - "http://172.22.6.3:32587/kiali/*" + "http://172.22.6.3:32588/kiali/*" ], "webOrigins": [ - "http://172.22.6.3:32587" + "http://172.22.6.3:32588" ], "notBefore": 0, "bearerOnly": false, @@ -1082,6 +1373,70 @@ "offline_access", "microprofile-jwt" ] + }, + { + "id": "af4f2785-a6fa-4bd5-9a28-469f1e513bd3", + "clientId": "tmaxcloud", + "name": "tmaxcloud", + "description": "tmaxcloud portal test", + "rootUrl": "", + "adminUrl": "", + "baseUrl": "", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "secret": "**********", + "redirectUris": [ + "http://192.168.8.*", + "https://192.168.8.*" + ], + "webOrigins": [ + "*" + ], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": true, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": false, + "publicClient": true, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "saml.assertion.signature": "false", + "access.token.lifespan": "60", + "saml.multivalued.roles": "false", + "saml.force.post.binding": "false", + "saml.encrypt": "false", + "login_theme": "tmax", + "saml.server.signature": "false", + "saml.server.signature.keyinfo.ext": "false", + "exclude.session.state.from.auth.response": "false", + "saml_force_name_id_format": "false", + "saml.client.signature": "false", + "tls.client.certificate.bound.access.tokens": "false", + "saml.authnstatement": "false", + "display.on.consent.screen": "false", + "saml.onetimeuse.condition": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "defaultClientScopes": [ + "web-origins", + "role_list", + "profile", + "roles", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "offline_access", + "microprofile-jwt" + ] } ], "clientScopes": [ @@ -1355,6 +1710,29 @@ } ] }, + { + "id": "6dd51ae7-2b85-461f-ad48-03b798c15b08", + "name": "gatekeeper", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "1fd5a122-0cc7-44ab-82ed-385362042b33", + "name": "gatekeeper", + "protocol": "openid-connect", + "protocolMapper": "oidc-audience-mapper", + "consentRequired": false, + "config": { + "included.client.audience": "gatekeeper", + "id.token.claim": "false", + "access.token.claim": "true" + } + } + ] + }, { "id": "4c7107dc-6fca-4dbf-8af1-8767533b0e40", "name": "email", @@ -1632,15 +2010,7 @@ "phone", "microprofile-jwt" ], - "browserSecurityHeaders": { - "contentSecurityPolicyReportOnly": "", - "xContentTypeOptions": "nosniff", - "xRobotsTag": "none", - "xFrameOptions": "SAMEORIGIN", - "contentSecurityPolicy": "frame-src 'self'; frame-ancestors 'self'; object-src 'none';", - "xXSSProtection": "1; mode=block", - "strictTransportSecurity": "max-age=31536000; includeSubDomains" - }, + "browserSecurityHeaders": {}, "smtpServer": { "host": "mail.tmax.co.kr", "from": "taegeon_woo@tmax.co.kr", @@ -1880,7 +2250,7 @@ "ko", "en" ], - "defaultLocale": "ko", + "defaultLocale": "en", "authenticationFlows": [ { "id": "b6231713-264b-4596-a39b-a5c3070c8efa", @@ -2033,6 +2403,69 @@ } ] }, + { + "id": "0f96b55a-06c0-4228-a821-e8c5dc574db0", + "alias": "Registration with SMS", + "description": "registration flow", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": false, + "authenticationExecutions": [ + { + "authenticator": "registration-page-form", + "requirement": "REQUIRED", + "priority": 10, + "flowAlias": "Registration with SMS registration form", + "userSetupAllowed": false, + "autheticatorFlow": true + } + ] + }, + { + "id": "06410df8-4a19-4020-a38c-7bbe9ed101c3", + "alias": "Registration with SMS registration form", + "description": "registration form", + "providerId": "form-flow", + "topLevel": false, + "builtIn": false, + "authenticationExecutions": [ + { + "authenticator": "registration-user-creation", + "requirement": "REQUIRED", + "priority": 20, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "registration-profile-action", + "requirement": "DISABLED", + "priority": 40, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "tmax-phone-validation", + "requirement": "REQUIRED", + "priority": 50, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "registration-password-action", + "requirement": "REQUIRED", + "priority": 60, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "registration-recaptcha-action", + "requirement": "DISABLED", + "priority": 61, + "userSetupAllowed": false, + "autheticatorFlow": false + } + ] + }, { "id": "e0b012d1-ff3e-4d7c-9030-d612350ae884", "alias": "Reset - Conditional OTP", @@ -2144,6 +2577,92 @@ } ] }, + { + "id": "6c31c695-f95a-4345-96c8-b0102d7d7bca", + "alias": "browser with SMS", + "description": "browser based authentication", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": false, + "authenticationExecutions": [ + { + "authenticator": "auth-cookie", + "requirement": "ALTERNATIVE", + "priority": 10, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "auth-spnego", + "requirement": "DISABLED", + "priority": 20, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "identity-provider-redirector", + "requirement": "ALTERNATIVE", + "priority": 25, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "requirement": "ALTERNATIVE", + "priority": 30, + "flowAlias": "browser with SMS forms", + "userSetupAllowed": false, + "autheticatorFlow": true + } + ] + }, + { + "id": "6ee192fc-c4ad-4b2f-b20a-6a46bad8b136", + "alias": "browser with SMS Browser - Conditional OTP", + "description": "Flow to determine if the OTP is required for the authentication", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": false, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "requirement": "REQUIRED", + "priority": 10, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "authenticator": "auth-otp-form", + "requirement": "REQUIRED", + "priority": 20, + "userSetupAllowed": false, + "autheticatorFlow": false + } + ] + }, + { + "id": "c2b0c6b0-a777-401f-bc9e-111a005303ad", + "alias": "browser with SMS forms", + "description": "Username, password, otp and other auth forms.", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": false, + "authenticationExecutions": [ + { + "authenticator": "auth-username-password-form", + "requirement": "REQUIRED", + "priority": 10, + "userSetupAllowed": false, + "autheticatorFlow": false + }, + { + "requirement": "DISABLED", + "priority": 20, + "flowAlias": "browser with SMS Browser - Conditional OTP", + "userSetupAllowed": false, + "autheticatorFlow": true + } + ] + }, { "id": "ce345a5f-5342-490e-8d44-c4c17afc4da5", "alias": "clients", @@ -2338,7 +2857,7 @@ }, { "authenticator": "registration-profile-action", - "requirement": "REQUIRED", + "requirement": "DISABLED", "priority": 40, "userSetupAllowed": false, "autheticatorFlow": false @@ -2472,7 +2991,7 @@ "alias": "VERIFY_EMAIL", "name": "Verify Email", "providerId": "VERIFY_EMAIL", - "enabled": true, + "enabled": false, "defaultAction": false, "priority": 50, "config": {} diff --git a/HyperAuth/manifest/tmaxRealmImport.sh b/HyperAuth/manifest/tmaxRealmImport.sh index 9fd150c2..b703ff40 100644 --- a/HyperAuth/manifest/tmaxRealmImport.sh +++ b/HyperAuth/manifest/tmaxRealmImport.sh @@ -37,7 +37,7 @@ curl -g -i -X POST \ "department": "tmaxCloud", "description": "hypercloud admin" }, - "username": "admin-tmax.co.kr", + "username": "admin@tmax.co.kr", "emailVerified": "", "email": "hc-admin@tmax.co.kr" }' \ @@ -55,4 +55,4 @@ curl 'http://'$hyperauthserver':8080/auth/admin/realms/tmax/users/'$userid'/rese -X 'PUT' \ -H "authorization: Bearer $token" \ -H "content-type: application/json;charset=UTF-8" \ - --data-binary '{"type":"password","value":"admin","temporary":false}' + --data-binary '{"type":"password","value":"Tmaxadmin1!","temporary":false}' diff --git a/HyperCloud Operator/v4.1.0.41/README.md b/HyperCloud Operator/v4.1.0.41/README.md index 8714a254..9c21b28f 100644 --- a/HyperCloud Operator/v4.1.0.41/README.md +++ b/HyperCloud Operator/v4.1.0.41/README.md @@ -73,22 +73,23 @@ $ sudo docker push ${REGISTRY}/tmaxcloudck/hypercloud-operator:b${HPCD_VERSION} ``` -3. [Nginx Ingress Controller 설치 가이드]([https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/IngressNginx/README.md](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/IngressNginx/README.md)를 참고하여 Nginx Ingress Controller 설치에 필요한 이미지 준비 +3. [Nginx Ingress Controller 설치 가이드]([https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/shared/README.md](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/shared/README.md)를 참고하여 Nginx Ingress Controller 설치에 필요한 이미지 준비 * 비고: - * [폐쇄망 설치 가이드]([https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/IngressNginx/README.md#%ED%8F%90%EC%87%84%EB%A7%9D-%EC%84%A4%EC%B9%98-%EA%B0%80%EC%9D%B4%EB%93%9C](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/IngressNginx/README.md#%ED%8F%90%EC%87%84%EB%A7%9D-%EC%84%A4%EC%B9%98-%EA%B0%80%EC%9D%B4%EB%93%9C)에서 아래의 부분 수행 + * [폐쇄망 설치 가이드]([https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/shared/README.md#%ED%8F%90%EC%87%84%EB%A7%9D-%EC%84%A4%EC%B9%98-%EA%B0%80%EC%9D%B4%EB%93%9C](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/shared/README.md#%ED%8F%90%EC%87%84%EB%A7%9D-%EC%84%A4%EC%B9%98-%EA%B0%80%EC%9D%B4%EB%93%9C)에서 아래의 부분 수행 * 1-1. 작업 디렉토리 생성 및 환경 설정 * 1-2. 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다. * 2-1. 위의 과정에서 생성한 tar 파일들을 폐쇄망 환경으로 이동시킨 뒤 사용하려는 registry에 이미지를 push한다. ## Install Steps -1. [1.initialization.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-1-1initializationyaml-%EC%8B%A4%ED%96%89) -2. [CRD 적용](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-2-crd-%EC%A0%81%EC%9A%A9) -3. [2.mysql-settings.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-3-2mysql-settingsyaml-%EC%8B%A4%ED%96%89) -4. [3.mysql-create.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-4-3mysql-createyaml-%EC%8B%A4%ED%96%89) -5. [6.hypercloud4-operator.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-5-6hypercloud4-operatoryaml-%EC%8B%A4%ED%96%89) -6. [8.default-auth-object-init.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-6-8default-auth-object-inityaml-%EC%8B%A4%ED%96%89) -7. [webhook-config 설정](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-7-webhook-config-%EC%84%A4%EC%A0%95) -8. [7.nginx-controller.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.41/README.md#step-8-9nginx-controlleryaml-%EC%8B%A4%ED%96%89) +0. [install yaml 수정](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-0-install--yaml-%EC%88%98%EC%A0%95) +1. [1.initialization.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-1-1initializationyaml-%EC%8B%A4%ED%96%89) +2. [CRD 적용](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-2-crd-%EC%A0%81%EC%9A%A9) +3. [2.mysql-settings.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-3-2mysql-settingsyaml-%EC%8B%A4%ED%96%89) +4. [3.mysql-create.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-4-3mysql-createyaml-%EC%8B%A4%ED%96%89) +5. [6.hypercloud4-operator.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-5-6hypercloud4-operatoryaml-%EC%8B%A4%ED%96%89) +6. [8.default-auth-object-init.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-6-8default-auth-object-inityaml-%EC%8B%A4%ED%96%89) +7. [webhook-config 설정](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-7-webhook-config-%EC%84%A4%EC%A0%95) +8. [7.nginx-controller.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.0.41/README.md#step-8-7nginx-controlleryaml-%EC%8B%A4%ED%96%89) ## Step 0. install yaml 수정 @@ -121,6 +122,7 @@ $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/UsergroupCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/TokenCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/ClientCRD.yaml + $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/clusterMenuPolicyCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/UserSecurityPolicyCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Claim/NamespaceClaimCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Claim/ResourceQuotaClaimCRD.yaml diff --git a/HyperCloud Operator/v4.1.1.0/README.md b/HyperCloud Operator/v4.1.1.0/README.md index 5762a5a1..0786dd6b 100644 --- a/HyperCloud Operator/v4.1.1.0/README.md +++ b/HyperCloud Operator/v4.1.1.0/README.md @@ -75,7 +75,7 @@ 1. Nginx Ingress Controller 설치 * 목적: Hypercloud Operator 내 기능(Reigstry Operator) 사용 * [Nginx Ingress Controller 설치 가이드] - * [https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/README.md](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/README.md) + * [https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/shared/README.md](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/shared/README.md) 2. Secret Watcher 설치 * 목적: Hypercloud Operator 내 기능(Reigstry Operator) 사용 * [secret-watcher 설치 가이드] @@ -87,6 +87,7 @@ 3. [2.mysql-settings.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.1.0/README.md#step-3-2mysql-settingsyaml-%EC%8B%A4%ED%96%89) 4. [3.mysql-create.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.1.0/README.md#step-4-3mysql-createyaml-%EC%8B%A4%ED%96%89) 5. [4.hypercloud4-operator.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.1.0/README.md#step-5-4hypercloud4-operatoryaml-%EC%8B%A4%ED%96%89) +6. [6.default-auth-object-init.yaml 실행](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.1.0/README.md#step-6-6default-auth-object-inityaml-%EC%8B%A4%ED%96%89) ## Step 0. install yaml 수정 @@ -100,7 +101,7 @@ $ sed -i 's/mysql:5.6/'${REGISTRY}'\/mysql:5.6/g' ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_Install/3.mysql-create.yaml $ sed -i 's/tmaxcloudck\/hypercloud-operator/'${REGISTRY}'\/tmaxcloudck\/hypercloud-operator/g' ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_Install/4.hypercloud4-operator.yaml - $ sed -i 's/{HPCD_VERSION}/'${HPCD_VERSION}'/g' ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_Install/4.hypercloud4-operator.yaml + $ sed -i 's/{HPCD_VERSION}/'b${HPCD_VERSION}'/g' ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_Install/4.hypercloud4-operator.yaml ``` @@ -115,6 +116,7 @@ * 목적 : `hypercloud crd 생성` * 실행 : *CRD.yaml실행 ```bash + $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/clusterMenuPolicyCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Auth/UserSecurityPolicyCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Claim/NamespaceClaimCRD.yaml $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_CRD/${HPCD_VERSION}/Claim/ResourceQuotaClaimCRD.yaml @@ -149,3 +151,10 @@ ```bash $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_Install/4.hypercloud4-operator.yaml ``` + +## Step 6. 6.default-auth-object-init.yaml 실행 +* 목적: `admin ClusterRoleBinding 및 namespace-owner, namespace-user, clusterrole-trial, clusterrole-new-user, namespace-listget ClusterRole 생성` +* 실행: + ```bash + $ kubectl apply -f ${HPCD_HOME}/hypercloud-operator-${HPCD_VERSION}/_yaml_Install/6.default-auth-object-init.yaml + ``` diff --git a/HyperCloud Webhook/README.md b/HyperCloud Webhook/README.md index a77a6042..55e5f1db 100644 --- a/HyperCloud Webhook/README.md +++ b/HyperCloud Webhook/README.md @@ -53,7 +53,6 @@ * 생성 순서 : 아래의 command를 실행하여 사용하고자 하는 image 버전을 수정한다. ([02_webhook-deployment.yaml](manifests/02_webhook-deployment.yaml)) ```bash $ sed -i 's/{webhook_version}/'${WEBHOOK_VERSION}'/g' 02_webhook-deployment.yaml - $ sed -i 's/{hostname}/'${HOSTNAME}'/g' 02_webhook-deployment.yaml ``` * 비고 : * 폐쇄망에서 설치를 진행하여 별도의 image registry를 사용하는 경우 registry 정보를 추가로 설정해준다. @@ -83,6 +82,7 @@ ## Step 5. HyperCloud Audit Webhook Config 생성 * 목적 : `앞서 생성한 인증서 정보를 기반으로 Audit Webhook 연동 설정 파일 생성` * 생성 순서 : 아래의 command를 실행하여 Webhook Config를 생성한다. ([05_gen-audit-config.sh](manifests/05_gen-audit-config.sh)) +* 주의: 마스터 다중화일 경우 cp 명령어를 모든 마스터에서 진행한다. ```bash $ sh 05_gen-audit-config.sh $ cp 06_audit-webhook-config /etc/kubernetes/pki/audit-webhook-config @@ -92,6 +92,7 @@ ## Step 6. HyperCloud Audit Webhook Config 적용 * 목적 : `Audit Webhook 연동 설정을 적용하여 API 서버가 Audit Webhook Server와 HTTPS 통신을 하도록 설정` * 생성 순서 : /etc/kubernetes/manifests/kube-apiserver.yaml을 아래와 같이 수정한다. +* 주의: 마스터 다중화일 경우 모든 마스터에서 진행한다. ``` spec.containers.command: - --audit-log-path=/var/log/kubernetes/apiserver/audit.log @@ -104,6 +105,7 @@ ## Step 7. test-yaml 배포 * 목적 : `Webhook Server 동작 검증` * 생성 순서 : [namespaceclaim.yaml](manifests/test-yaml/namespaceclaim.yaml) 실행 `ex) kubectl apply -f namespaceclaim.yaml` +* 주의: 마스터 다중화일 경우 모든 마스터에서 진행한다. ``` kubectl describe namespaceclaim example-namespace-webhook Annotation에 creator/updater/createdTime/updatedTime 필드가 생성 되었는지 확인한다. diff --git a/HyperCloud Webhook/manifests/02_webhook-deployment.yaml b/HyperCloud Webhook/manifests/02_webhook-deployment.yaml index cd0695f3..92d7bdb0 100644 --- a/HyperCloud Webhook/manifests/02_webhook-deployment.yaml +++ b/HyperCloud Webhook/manifests/02_webhook-deployment.yaml @@ -15,8 +15,6 @@ spec: labels: hypercloud4: webhook spec: - nodeSelector: - kubernetes.io/hostname: {hostname} containers: - name: hypercloud4-webhook image: tmaxcloudck/hypercloud-webhook:{webhook_version} diff --git a/IngressNginx/shared/README.md b/IngressNginx/shared/README.md index 6e445f52..e9a859f2 100644 --- a/IngressNginx/shared/README.md +++ b/IngressNginx/shared/README.md @@ -19,6 +19,10 @@ $ export INGRESS_CLASS=nginx-shd $ export NGINX_INGRESS_VERSION=0.33.0 $ export KUBE_WEBHOOK_CERTGEN_VERSION=v1.2.2 + + # image를 push할 폐쇄망 Registry 주소 입력(예:192.168.6.150:5000) + $ export REGISTRY= + $ cd $NGINX_INGRESS_HOME ``` * 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다. @@ -47,8 +51,8 @@ ## Install Steps -0. [deploy yaml 수정](https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/IngressNginx#step0-deploy-yaml-%EC%88%98%EC%A0%95) -1. [Nginx Ingress Controller 배포](https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/IngressNginx#step-1-nginx-ingress-controller-%EB%B0%B0%ED%8F%AC) +0. [deploy yaml 수정](https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/IngressNginx/shared#step0-deploy-yaml-%EC%88%98%EC%A0%95) +1. [Nginx Ingress Controller 배포](https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/IngressNginx/shared#step-1-nginx-ingress-controller-%EB%B0%B0%ED%8F%AC) ## Step0. deploy yaml 수정 diff --git a/IngressNginx/system/README.md b/IngressNginx/system/README.md index e1df3129..046263d4 100644 --- a/IngressNginx/system/README.md +++ b/IngressNginx/system/README.md @@ -17,6 +17,10 @@ $ export NGINX_INGRESS_HOME=~/install-ingress-nginx-system $ export NGINX_INGRESS_VERSION=0.33.0 $ export KUBE_WEBHOOK_CERTGEN_VERSION=v1.2.2 + + # image를 push할 폐쇄망 Registry 주소 입력(예:192.168.6.150:5000) + $ export REGISTRY= + $ cd $NGINX_INGRESS_HOME ``` * 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다. diff --git a/Istio/README.md b/Istio/README.md index 5faa9d77..91507f3c 100644 --- a/Istio/README.md +++ b/Istio/README.md @@ -228,3 +228,5 @@ * http://$PRODUCTPAGE_URL/productpage 에 접속해 정상적으로 배포되었는지 확인한 뒤, kiali dashboard(http://$KIALI_URL/kiali)에 접속해 아래 그림과 같이 서비스간에 관계를 표현해주는 그래프가 나오는지 확인한다. ![image](figure/bookinfo-example.png) + + diff --git a/Istio/yaml/kiali-ingress.yaml b/Istio/yaml/kiali-ingress.yaml index 3ea581d1..71134955 100644 --- a/Istio/yaml/kiali-ingress.yaml +++ b/Istio/yaml/kiali-ingress.yaml @@ -3,6 +3,8 @@ kind: Ingress metadata: name: kiali-ingress namespace: istio-system + annotations: + kubernetes.io/ingress.class: nginx-system spec: tls: - secretName: kiali-https-secret diff --git a/K8S_Master/KUBE_CERTIFICATE_UPDATE_README.md b/K8S_Master/KUBE_CERTIFICATE_UPDATE_README.md new file mode 100644 index 00000000..c2c2a846 --- /dev/null +++ b/K8S_Master/KUBE_CERTIFICATE_UPDATE_README.md @@ -0,0 +1,90 @@ +# kubeadm을 이용한 인증서 갱신 + +## Prerequisites + * kubeadm + +## Step0. 인증서 만료 확인 +* 목적 : `인증서가 만료되는 시기를 확인한다.` +* 순서 : + ```bash + kubeadm alpha certs check-expiration + ``` + ```bash + CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED + admin.conf Nov 30, 2021 07:23 UTC 364d no + apiserver Nov 30, 2021 07:23 UTC 364d ca no + apiserver-etcd-client Nov 30, 2021 07:23 UTC 364d etcd-ca no + apiserver-kubelet-client Nov 30, 2021 07:23 UTC 364d ca no + controller-manager.conf Nov 30, 2021 07:23 UTC 364d no + etcd-healthcheck-client Nov 30, 2021 07:23 UTC 364d etcd-ca no + etcd-peer Nov 30, 2021 07:23 UTC 364d etcd-ca no + etcd-server Nov 30, 2021 07:23 UTC 364d etcd-ca no + front-proxy-client Nov 30, 2021 07:23 UTC 364d front-proxy-ca no + scheduler.conf Nov 30, 2021 07:23 UTC 364d no + + CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED + ca Nov 21, 2030 06:29 UTC 9y no + etcd-ca Nov 21, 2030 06:29 UTC 9y no + front-proxy-ca Nov 21, 2030 06:29 UTC 9y no + ``` + +## Step1. 인증서 갱신 +* 목적 : `인증서를 수동으로 갱신한다.` +* 순서 : + * 아래 명령어는 /etc/kubernetes/pki 에 저장된 CA(또는 프론트 프록시 CA) 인증서와 키를 사용하여 갱신을 수행한다. + * kubeadm으로 생성된 클라이언트 인증서는 1년 기준이다. + ```bash + kubeadm alpha certs renew all + + + [renew] Reading configuration from the cluster... + [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' + + certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed + certificate for serving the Kubernetes API renewed + certificate the apiserver uses to access etcd renewed + certificate for the API server to connect to kubelet renewed + certificate embedded in the kubeconfig file for the controller manager to use renewed + certificate for liveness probes to healthcheck etcd renewed + certificate for etcd nodes to communicate with each other renewed + certificate for serving etcd renewed + certificate for the front proxy client renewed + certificate embedded in the kubeconfig file for the scheduler manager to use renewed + ``` + * kube-system pod (kube scheduler, api server, controller, etcd) 재기동 + ```bash + kubectl get pod -n kube-system -o yaml | kubectl replace --force -f- + + ex) kubectl get pod kube-apiserver-k8s-master -n kube-system -o yaml | kubectl replace --force -f- + ``` + * config 복사 + ```bash + cp -i /etc/kubernetes/admin.conf /root/.kube/config + ``` +* 비고 : + * warning : 다중화 클러스터 구성의 경우, 모든 컨트롤 플레인 노드에서 이 명령을 실행해야 한다. + * 이미 인증서가 만료된 경우 아래 가이드를 참조하여 인증서를 갱신한다. + * 기존 인증서 및 config 백업 (권장사항) + ```bash + mkdir ~/cert_temp + + cd /etc/kubernetes/pki/ + mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/cert_temp + + cd /etc/kubernetes/pki/etcd + mv {ca.crt,ca.key,healthcheck-client.crt,healthcheck-client.key,peer.crt,peer.key,server.crt,server.key} ~/cert_temp + + cd /etc/kubernetes/ + mv {admin.conf,controller-manager.conf,mv kubelet.conf,scheduler.conf} ~/cert_temp + ``` + * 새 인증서 생성 및 config 변경 적용 + ```bash + kubeadm init phase certs all --apiserver-advertise-address + + kubeadm init phase kubeconfig all + ``` + * reboot + * kube config 복사 + ```bash + cp -i /etc/kubernetes/admin.conf $HOME/.kube/config + ``` diff --git a/K8S_Master/KUBE_VERSION_UPGRADE_README.md b/K8S_Master/KUBE_VERSION_UPGRADE_README.md index 8159ca16..cd641be9 100644 --- a/K8S_Master/KUBE_VERSION_UPGRADE_README.md +++ b/K8S_Master/KUBE_VERSION_UPGRADE_README.md @@ -11,9 +11,118 @@ * 하나의 MINOR 버전에서 다음 MINOR 버전으로, 또는 동일한 MINOR의 PATCH 버전 사이에서만 업그레이드할 수 있다. * 즉, 업그레이드할 때 MINOR 버전을 건너 뛸 수 없다. 예를 들어, 1.y에서 1.y+1로 업그레이드할 수 있지만, 1.y에서 1.y+2로 업그레이드할 수는 없다. * ex) 1.15 버전에서 1.17 버전으로 한번에 업그레이드는 불가능 하다. 1.15 -> 1.16 -> 1.17 스텝을 진행 해야 한다. -* upgrade시 처음 구성한 클러스터 container runtime 변경(docker -> crio)은 불가능하다. * runtime으로 crio 사용시, CRI-O 메이저와 마이너 버전은 쿠버네티스 메이저와 마이너 버전이 일치해야 한다. 따라서 업데이트한 쿠버네티스 버전에 따라 crio 버전도 함께 업데이트 한다. +## 폐쇄망 가이드 +1. **폐쇄망에서 설치하는 경우** 아래 가이드를 참고 하여 image registry를 먼저 구축한다. + * https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/Image_Registry +2. 사용하는 image repository에 k8s 설치 시 필요한 이미지를 push한다. + * 작업 디렉토리 생성 및 환경 설정 + ```bash + $ mkdir -p ~/k8s-install + $ cd ~/k8s-install + ``` + * 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다. (1.15.x -> 1.17.x으로 upgrade 하는 경우 두 버전의 image들이 모두 필요하다) + * v1.16.15 images + ```bash + $ sudo docker pull k8s.gcr.io/kube-proxy:v1.16.15 + $ sudo docker pull k8s.gcr.io/kube-apiserver:v1.16.15 + $ sudo docker pull k8s.gcr.io/kube-controller-manager:v1.16.15 + $ sudo docker pull k8s.gcr.io/kube-scheduler:v1.16.15 + $ sudo docker pull k8s.gcr.io/etcd:3.3.15-0 + $ sudo docker pull k8s.gcr.io/coredns:1.6.2 + $ sudo docker pull k8s.gcr.io/pause:3.1 + ``` + * v1.17.6 images + ```bash + $ sudo docker pull k8s.gcr.io/kube-proxy:v1.17.6 + $ sudo docker pull k8s.gcr.io/kube-apiserver:v1.17.6 + $ sudo docker pull k8s.gcr.io/kube-controller-manager:v1.17.6 + $ sudo docker pull k8s.gcr.io/kube-scheduler:v1.17.6 + $ sudo docker pull k8s.gcr.io/etcd:3.4.3-0 + $ sudo docker pull k8s.gcr.io/coredns:1.6.5 + $ sudo docker pull k8s.gcr.io/pause:3.1 + ``` + + * docker image를 tar로 저장한다. + * v1.16.15 images + ```bash + $ sudo docker save -o kube-proxy-1.16.tar k8s.gcr.io/kube-proxy:v1.16.15 + $ sudo docker save -o kube-controller-manager-1.16.tar k8s.gcr.io/kube-apiserver:v1.16.15 + $ sudo docker save -o etcd-1.16.tar k8s.gcr.io/etcd:3.3.15-0 + $ sudo docker save -o coredns-1.16.tar k8s.gcr.io/coredns:1.6.2 + $ sudo docker save -o kube-scheduler-1.16.tar k8s.gcr.io/kube-scheduler:v1.16.15 + $ sudo docker save -o kube-apiserver-1.16.tar k8s.gcr.io/kube-apiserver:v1.16.15 + $ sudo docker save -o pause-1.16.tar k8s.gcr.io/pause:3.1 + ``` + * v1.17.6 images + ```bash + $ sudo docker save -o kube-proxy-1.17.tar k8s.gcr.io/kube-proxy:v1.17.6 + $ sudo docker save -o kube-controller-manager-1.17.tar k8s.gcr.io/kube-controller-manager:v1.17.6 + $ sudo docker save -o etcd-1.17.tar k8s.gcr.io/etcd:3.4.3-0 + $ sudo docker save -o coredns-1.17.tar k8s.gcr.io/coredns:1.6.5 + $ sudo docker save -o kube-scheduler-1.17.tar k8s.gcr.io/kube-scheduler:v1.17.6 + $ sudo docker save -o kube-apiserver-1.17.tar k8s.gcr.io/kube-apiserver:v1.17.6 + $ sudo docker save -o pause-1.17.tar k8s.gcr.io/pause:3.1 + ``` +3. 위의 과정에서 생성한 tar 파일들을 폐쇄망 환경으로 이동시킨 뒤 사용하려는 registry에 이미지를 push한다. + * v1.16.15 images + ```bash + $ sudo docker load -i kube-apiserver-1.16.tar + $ sudo docker load -i kube-scheduler-1.16.tar + $ sudo docker load -i kube-controller-manager-1.16.tar + $ sudo docker load -i kube-proxy-1.16.tar + $ sudo docker load -i etcd-1.16.tar + $ sudo docker load -i coredns-1.16.tar + $ sudo docker load -i pause-1.16.tar + ``` + ```bash + $ sudo docker tag k8s.gcr.io/kube-apiserver:v1.16.15 ${REGISTRY}/k8s.gcr.io/kube-apiserver:v1.16.15 + $ sudo docker tag k8s.gcr.io/kube-proxy:v1.16.15 ${REGISTRY}/k8s.gcr.io/kube-proxy:v1.16.15 + $ sudo docker tag k8s.gcr.io/kube-controller-manager:v1.16.15 ${REGISTRY}/k8s.gcr.io/kube-controller-manager:v1.16.15 + $ sudo docker tag k8s.gcr.io/etcd:3.3.15-0 ${REGISTRY}/k8s.gcr.io/etcd:3.3.15-0 + $ sudo docker tag k8s.gcr.io/coredns:1.6.2 ${REGISTRY}/k8s.gcr.io/coredns:1.6.2 + $ sudo docker tag k8s.gcr.io/kube-scheduler:v1.16.15 ${REGISTRY}/k8s.gcr.io/kube-scheduler:v1.16.15 + $ sudo docker tag k8s.gcr.io/pause:3.1 ${REGISTRY}/k8s.gcr.io/pause:3.1 + ``` + ```bash + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-apiserver:v1.16.15 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-proxy:v1.16.15 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-controller-manager:v1.16.15 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/etcd:3.3.15-0 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/coredns:1.6.2 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-scheduler:v1.16.15 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/pause:3.1 + ``` + * v1.17.6 images + ```bash + $ sudo docker load -i kube-apiserver-1.17.tar + $ sudo docker load -i kube-scheduler-1.17.tar + $ sudo docker load -i kube-controller-manager-1.17.tar + $ sudo docker load -i kube-proxy-1.17.tar + $ sudo docker load -i etcd-1.17.tar + $ sudo docker load -i coredns-1.17.tar + $ sudo docker load -i pause-1.17.tar + ``` + ```bash + $ sudo docker tag k8s.gcr.io/kube-apiserver:v1.17.6 ${REGISTRY}/k8s.gcr.io/kube-apiserver:v1.17.6 + $ sudo docker tag k8s.gcr.io/kube-proxy:v1.17.6 ${REGISTRY}/k8s.gcr.io/kube-proxy:v1.17.6 + $ sudo docker tag k8s.gcr.io/kube-controller-manager:v1.17.6 ${REGISTRY}/k8s.gcr.io/kube-controller-manager:v1.17.6 + $ sudo docker tag k8s.gcr.io/etcd:3.4.3-0 ${REGISTRY}/k8s.gcr.io/etcd:3.4.3-0 + $ sudo docker tag k8s.gcr.io/coredns:1.6.5 ${REGISTRY}/k8s.gcr.io/coredns:1.6.5 + $ sudo docker tag k8s.gcr.io/kube-scheduler:v1.17.6 ${REGISTRY}/k8s.gcr.io/kube-scheduler:v1.17.6 + $ sudo docker tag k8s.gcr.io/pause:3.1 ${REGISTRY}/k8s.gcr.io/pause:3.1 + ``` + ```bash + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-apiserver:v1.17.6 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-proxy:v1.17.6 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-controller-manager:v1.17.6 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/etcd:3.4.3-0 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/coredns:1.6.5 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/kube-scheduler:v1.17.6 + $ sudo docker push ${REGISTRY}/k8s.gcr.io/pause:3.1 + ``` + ## Steps 0. [master upgrade](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/K8S_Master/KUBE_VERSION_UPGRADE_README.md#step0-kubernetes-master-upgrade) 1. [node upgrade](https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/K8S_Master/KUBE_VERSION_UPGRADE_README.md#step1-kubernetes-node-upgrade) @@ -32,13 +141,35 @@ kubeadm version ``` * node drain - ```bash - kubectl drain --ignore-daemonsets + * node drain 전 체크 사항 + * PDB가 존재하는 Pod가 drain하려는 node에 생성되어있는 경우 evict가 제한 되기 때문에, 아래 명령어로 drain이 가능한 상태인지 확인한다. + ```bash + kubectl get pdb -A + or + kubectl get pdb -oyaml + ``` + * ALLOWED DISRUPTIONS 및 drain 시키려는 node의 pod 상태를 확인한다. + * PDB의 ALLOWED DISRUPTIONS가 drain을 시도하는 node에 뜬 pod(pdb 설정 pod) 개수보다 적을 경우 아래와 같이 다른 노드로 재스케줄링이 필요하다. + * ex) virt-api pod가 drain하려는 node에 2개 떠있는데, ALLOWED DISRUPTIONS는 0 또는 1일 경우 + * 해당 조건에 만족하지 않는 경우 'Cannot evict pod as it would violate the pod's disruption budget' 와 같은 에러가 발생할 수 있다. + * 해결 방법 + * 1) 해당 Pod를 다른 Node로 재스케줄링을 시도한다. + ```bash + kubectl delete pod + ``` + * 2) 다른 Node의 리소스 부족, noScheduling 설정 등으로 인해 a번 재스케줄링이 불가할 경우엔 PDB 데이터를 삭제하고 drain한 후에 PDB 데이터를 복구한다. + ```bash + kubectl get pdb -o yaml > pdb-backup.yaml + kubectl drain --ignore-daemonsets --delete-local-data + kubectl apply -f pdb-backup.yaml + ``` + * node drain 실행 + * warning: node drain시 해당 node상의 pod가 evict되기 때문에, pod의 local-data의 경우 보존되지 않음 + ```bash + kubectl drain --ignore-daemonsets --delete-local-data - ex) kubectl drain k8s-master --ignore-daemonsets - ``` - * node drain시 cannot delete Pods with local storage (use --delete-local-data to override) 문구가 보인 경우 - * 기존 VM의 emptydir에 작업 내용이 필요한 경우 기존 etcd 백업을 한다. + ex) kubectl drain k8s-master --ignore-daemonsets --delete-local-data + ``` * 업그레이드 plan 변경 ```bash sudo kubeadm upgrade plan @@ -147,7 +278,7 @@ ```bash sudo systemctl daemon-reload sudo systemctl restart kubelet - ``` + ``` * 비고 : * master 다중화 구성 클러스터 업그레이드 시에는 다음과 같은 명령어를 실행한다. * 첫번째 컨트롤 플레인 업그레이드 시에는 위에 step을 진행하고, 나머지 컨트롤 플레인 업그레이드 시에는 아래의 명령어를 실행한다. @@ -164,13 +295,15 @@ ```bash kubeadm version ``` - * node drain - ```bash - kubectl drain --ignore-daemonsets - - ex) kubectl drain k8s-master2 --ignore-daemonsets - ``` - + * node drain + * 추가 컨트롤 플레인에서도 첫번째 컨트롤 플레인 node drain 전 체크 사항을 참고하여 drain 가능한 상태인지 체크한다. + * node drain 실행 + * node drain시 해당 node상의 pod가 evict되기 때문에, pod의 local-data의 경우 보존되지 않음 + ```bash + kubectl drain --ignore-daemonsets --delete-local-data + + ex) kubectl drain k8s-master2 --ignore-daemonsets --delete-local-data + ``` * 추가 컨트롤 프레인에서는 해당 명령어를 실행하지 않는다. (sudo kubeadm upgrade plan) * sudo kubeadm upgrade apply 명령어 대신에 sudo kubeadm upgrade node 명령어를 실행한다. ```bash @@ -193,7 +326,30 @@ sudo systemctl daemon-reload sudo systemctl restart kubelet ``` - + * 업그레이드 후 노드가 ready -> not ready 상태로 바뀐 경우 + * Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource + ```bash + sudo vi /var/lib/kubelet/config.yaml에 아래 옵션 추가 + + featureGates: + CSIMigration: false + + sudo systemctl restart kubelet + ``` + * 업그레이드시 runtime 변경을 하는 경우 (docker -> cri-o) + * crio 설치는 https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/K8S_Master#step-1-cri-o-%EC%84%A4%EC%B9%98를 참조한다. + ```bash + systemctl stop kubelet + + sudo vi /var/lib/kubelet/kubeadm-flags.env에 옵션 변경 + + 기존 (docker) : KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 + 변경 (cri-o) : KUBELET_KUBEADM_ARGS="--container-runtime=remote --cgroup-driver=systemd --container-runtime-endpoint=/var/run/crio/crio.sock" + + systemctl restart kubelet + systemctl restart docker ( #docker image registry node는 systemctl restart docker 명령어를 실행한다. ) + ``` + ## Step1. kubernetes node upgrade * 워커 노드의 업그레이드 절차는 워크로드를 실행하는 데 필요한 최소 용량을 보장하면서, 한 번에 하나의 노드 또는 한 번에 몇 개의 노드로 실행해야 한다. * 모든 worker node에서 kubeadm을 업그레이드한다. @@ -204,12 +360,36 @@ ex) (1.15.x-> 1.16.x) yum install -y kubeadm-1.17.x-0 --disableexcludes=kubernetes ``` -* 스케줄 불가능(unschedulable)으로 표시하고 워크로드를 축출하여 유지 보수할 노드를 준비한다. - ```bash - kubectl drain --ignore-daemonsets - ``` - * node drain시 cannot delete Pods with local storage (use --delete-local-data to override) 문구가 보인 경우 - * 기존 VM의 emptydir에 작업 내용이 필요한 경우 기존 etcd 백업을 한다. +* node drain + * node drain 전 체크 사항 + * PDB가 존재하는 Pod가 drain하려는 node에 생성되어있는 경우 evict가 제한 되기 때문에, 아래 명령어로 drain이 가능한 상태인지 확인한다. + ```bash + kubectl get pdb -A + or + kubectl get pdb -oyaml + ``` + * ALLOWED DISRUPTIONS 및 drain 시키려는 node의 pod 상태를 확인한다. + * PDB의 ALLOWED DISRUPTIONS가 drain을 시도하는 node에 뜬 pod(pdb 설정 pod) 개수보다 적을 경우 아래와 같이 다른 노드로 재스케줄링이 필요하다. + * ex) virt-api pod가 drain하려는 node에 2개 떠있는데, ALLOWED DISRUPTIONS는 0 또는 1일 경우 + * 해당 조건에 만족하지 않는 경우 'Cannot evict pod as it would violate the pod's disruption budget' 와 같은 에러가 발생할 수 있다. + * 해결 방법 + * 1) 해당 Pod를 다른 Node로 재스케줄링을 시도한다. + ```bash + kubectl delete pod + ``` + * 2) 다른 Node의 리소스 부족, noScheduling 설정 등으로 인해 a번 재스케줄링이 불가할 경우엔 PDB 데이터를 삭제하고 drain한 후에 PDB 데이터를 복구한다. + ```bash + kubectl get pdb -o yaml > pdb-backup.yaml + kubectl drain --ignore-daemonsets --delete-local-data + kubectl apply -f pdb-backup.yaml + ``` + * node drain 실행 + * warning : node drain시 해당 node상의 pod가 evict되기 때문에, pod의 local-data의 경우 보존되지 않음 + ```bash + kubectl drain --ignore-daemonsets --delete-local-data + + ex) kubectl drain k8s-node --ignore-daemonsets --delete-local-data + ``` * kubelet 구성 업그레이드 ```bash sudo kubeadm upgrade node @@ -231,3 +411,27 @@ ``` * 비고 : * 1.16.x -> 1.17.x로 업그레이드시 버전에 맞추어 위에 작업을 실행한다. + * 업그레이드 후 노드가 ready -> not ready 상태로 바뀐 경우 + * Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource + ```bash + sudo vi /var/lib/kubelet/config.yaml에 아래 옵션 추가 + + featureGates: + CSIMigration: false + + sudo systemctl restart kubelet + ``` + * 업그레이드시 runtime 변경을 하는 경우 (docker -> cri-o) + * crio 설치는 https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/K8S_Master#step-1-cri-o-%EC%84%A4%EC%B9%98를 참조한다. + ```bash + systemctl stop kubelet + + sudo vi /var/lib/kubelet/kubeadm-flags.env에 옵션 변경 + + 기존 (docker) : KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1" + 변경 (cri-o) : KUBELET_KUBEADM_ARGS="--container-runtime=remote --cgroup-driver=systemd --container-runtime-endpoint=/var/run/crio/crio.sock" + + systemctl restart kubelet + systemctl stop docker ( #docker image registry node는 systemctl restart docker 명령어를 실행한다. ) + + ``` diff --git a/K8S_Master/README.md b/K8S_Master/README.md index 56b2f60d..70c4cadb 100644 --- a/K8S_Master/README.md +++ b/K8S_Master/README.md @@ -172,15 +172,11 @@ sudo rm -rf /etc/cni/net.d/100-crio-bridge.conf sudo rm -rf /etc/cni/net.d/200-loopback.conf ``` - * 폐쇄망 환경에서 private registry 접근을 위해 crio.conf 내용을 수정한다. - * insecure_registry, registries, plugin_dirs 내용을 수정한다. - * sudo vi /etc/crio/crio.conf - * registries = ["{registry}:{port}" , "docker.io"] - * insecure_registries = ["{registry}:{port}"] - * plugin_dirs : "/opt/cni/bin" 추가 - * (폐쇄망) pause_image : "k8s.gcr.io/pause:3.1" 을 "{registry}:{port}/k8s.gcr.io/pause:3.1" 로 변경 + * crio.conf 내용을 수정한다. ( sudo vi /etc/crio/crio.conf ) + * plugin_dirs : "/opt/cni/bin" 추가 + * (폐쇄망) insecure_registries = ["{registry}:{port}"] + * (폐쇄망) pause_image : "k8s.gcr.io/pause:3.1" 을 "{registry}:{port}/k8s.gcr.io/pause:3.1" 로 변경 ![image](figure/crio_config.PNG) - * registries.conf 내용을 수정한다. * sudo vi /etc/containers/registries.conf ```bash diff --git a/K8S_Master/installer/yaml/kubeadm-config.yaml b/K8S_Master/installer/yaml/kubeadm-config.yaml index 3e709734..43eef9a2 100644 --- a/K8S_Master/installer/yaml/kubeadm-config.yaml +++ b/K8S_Master/installer/yaml/kubeadm-config.yaml @@ -13,7 +13,7 @@ controlPlaneEndpoint: {apiServer}:6443 imageRepository: {imageRegistry}/k8s.gcr.io networking: serviceSubnet: 10.96.0.0/16 - podSubnet: 10.244.0.0/16 + podSubnet: {podSubnet} --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration diff --git a/K8S_Master_LBNode/README.md b/K8S_Master_LBNode/README.md new file mode 100644 index 00000000..4484f413 --- /dev/null +++ b/K8S_Master_LBNode/README.md @@ -0,0 +1,358 @@ +# K8S Master 클러스터의 LBNode 설치 가이드(HAProxy + Keepalived) +* 본 가이드는 A, B로 구성. +* [A는 별도의 LBNode를 갖는 K8S 다중화 클러스터 구축을 위해 작성](#a-%EB%B3%84%EB%8F%84%EC%9D%98-lbnode%EB%A5%BC-%EA%B0%96%EB%8A%94-%EA%B2%BD%EC%9A%B0)되었음. + * 구축하려는 LBNode에 해당 파일들이 같은 디렉터리 내에 존재해야 함. + * LBNode 각각에서 아래의 작업들을 동일하게 수행해야 함. +* [B는 별도의 LBNode 없이, K8S 다중화 클러스터 내에서 HAProxy가 동작하도록 작성](#b-%EB%B3%84%EB%8F%84%EC%9D%98-lbnode%EB%A5%BC-%EA%B0%96%EC%A7%80-%EC%95%8A%EA%B3%A0-%ED%81%B4%EB%9F%AC%EC%8A%A4%ED%84%B0-%EB%82%B4%EC%97%90%EC%84%9C-haproxy%EB%A5%BC-%EB%8F%99%EC%9E%91%EC%8B%9C%ED%82%AC-%EA%B2%BD%EC%9A%B0)되었음. + * 구축하려는 MasterNode에 해당 파일들이 같은 디렉터리 내에 존재해야 함. + * MasterNode 각각에서 아래의 작업들을 동일하게 수행해야 함. +# A. 별도의 LBNode를 갖는 경우 +## 구성 요소 및 버전 +* Keepalived v1.3.5 # LBNode에 설치 +* HA-Proxy v1.5.18 # LBNode에 설치 +## Install Steps +0. 변수 설정 및 준비 +1. HAProxy + Keepalived 설치 +2. 설치한 서비스 기동 +3. K8S 클러스터 구축 + +## Step 0. 변수 설정 및 준비 +* 목적 : `LB Node 구축을 위한 변수 설정 및 준비과정` +* 순서 : + * Keepalived 와 HAProxy를 설치 및 동작시키기 위한 변수를 설정한다. + * 클러스터 구성에 사용할 각 Master Node의 IP, VIP, LBNode에 대한 정보를 입력한다. + ```bash + export MASTER1NAME=master1hostname # 클러스터로 구성할 Master Node의 host명을 각각 입력. + export MASTER2NAME=master2hostname + export MASTER3NAME=master3hostname + + export MASTER1IP=192.168.56.222 # Master Node의 IP를 각각 입력. + export MASTER2IP=192.168.56.223 + export MASTER3IP=192.168.56.224 + + export MASTERPORT=6443 # 기본적으로 Master Port는 6443을 사용. + export HAPROXYLBPORT=16443 # Master 와 동일한 Node에 설치시 반드시 MASTERPORT와 다른 Port를 사용해야 하며, 이경우 Master Join시에 이 변수로 설정한 Port를 사용해야 함. + + export LB1=192.168.56.250 # 현재 LB Node의 IP를 입력. + export LB2=192.168.56.249 # 다른 LB Node의 IP를 입력. + + export VIP=192.168.56.240 # K8S Master Join시VIP로 사용할 IP를 입력. + ``` + + * LB Node 구축을 위해 필요한 파일들을 동일한 위치에 다운로드 받는다. + ```bash + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/haproxy.cfg + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/keepalived.conf + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/lb_set_script.sh + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/notify_action.sh + ``` + + * SELinux 관련 플래그를 설정한다. + ```bash + sudo setsebool -P haproxy_connect_any=1 + ``` + + * LBNode에서 동작 중인 firewalld를 중지 및 비활성화 한다. + ```bash + sudo systemctl stop firewalld && sudo systemctl disable firewalld + ``` + + +## Step.1 HAProxy + Keepalived 설치 +* 목적 : `설치 스크립트를 실행하여 HAProxy와 Keepalived를 LBNode에 설치` +* 순서 : + * 설치 스크립트에 실행 권한을 주고, 실행한다. + ```bash + sudo chmod +x lb_set_script.sh + sudo ./lb_set_script.sh + ``` + + +## Step.2 설치한 서비스 기동 +* 목적 : `HAProxy와 Keepalived 기동` +* 순서 : + * 각 서비스의 설정파일에 Step0 에서 입력한 값들이 올바르게 설정되었는지 확인한다. + ```bash + sudo vi /etc/keepalived/keepalived.conf + sudo vi /etc/haproxy/haproxy.cfg + ``` + + * Keepalived 설정파일의 세부내용을 확인/수정한다. + * state 필드는 MASTER or BACKUP을 반드시 수정하며, priority 또한 수정한다. + * interface도 수정해줘야한다. + * unicast_src_ip 는 현재 설치 진행 중인 LB 서버(앞서 설정한 LB1 변수)이다. + * unicast_peer 는 다른 LB 서버(앞서 설정한 LB2 변수)이다. + ```bash + global_defs { + script_user root root + enable_script_security off + } + + vrrp_script chk_haproxy { + script "/usr/sbin/pidof haproxy" + interval 2 + weight 2 + } + + vrrp_instance VI_1 { + state MASTER # MASTER는 메인 LB, 백업 LB는 BACKUP 으로 설정 + interface enp0s8 # 사용할 interface + virtual_router_id 51 + priority 100 # MASTER의 우선순위를 가장 높게(ex. 100), BACKUP의 경우 그보다 낮게(ex. 99, 98) 설정. + advert_int 1 + authentication { # 인증에 사용될 password(동일하게 맞춰주기만 하면 됨) + auth_type PASS + auth_pass 1111 + } + + unicast_src_ip LB1 # LB 서버 local ip + + unicast_peer { + LB2 # 다른 LB 서버 local ip + } + + virtual_ipaddress { + VIP # 클러스터 구성에 사용될 VIP! + } + + notify_master "/bin/sh /etc/keepalived/notify_action.sh MASTER" + notify_backup "/bin/sh /etc/keepalived/notify_action.sh BACKUP" + notify_fault "/bin/sh /etc/keepalived/notify_action.sh FAULT" + notify_stop "/bin/sh /etc/keepalived/notify_action.sh STOP" + + track_script { + chk_haproxy + } + + track_interface { + enp0s8 # 사용할 interface + } + } + ``` + + * HA Proxy 설정파일의 세부내용을 확인/수정한다. + ```bash + global + log 127.0.0.1 local2 + maxconn 2000 + uid 0 + gid 0 + daemon # background process + + defaults + log global # global 설정 사용 + mode tcp # SSL 통신을 위해서는 TCP모드로 (http모드는 SSL 안됨) + option tcplog + option dontlognull # 데이터가 전송되지 않은 연결 로깅 제외 + retries 3 # 연결요청 재시도 횟수 + maxconn 2000 #option redispatch + timeout connect 10s + timeout client 1m + timeout server 1m + + frontend k8s-api + bind 0.0.0.0:HAPROXYLBPORT # Master Node와 동일 Node에 설치시, Master Join을 해당 port로 해야함. + default_backend k8s-api + + backend k8s-api + option tcp-check + balance roundrobin + server MASTER1NAME MASTER1IP:MASTERPORT check # Master 다중화 서버들 정보 기재 + server MASTER2NAME MASTER2IP:MASTERPORT check + server MASTER3NAME MASTER3IP:MASTERPORT check + ``` + + * 각 서비스를 활성화시켜주며 기동하고, 동작을 확인한다. + ```bash + sudo systemctl enable keepalived + sudo systemctl enable haproxy + + sudo systemctl daemon-reload + + sudo systemctl start keepalived + sudo systemctl start haproxy + + sudo systemctl status keepalived + sudo systemctl status haproxy + ``` + +## Step.3 K8S 클러스터 구축 +* 목적 : `LB Node 설정을 완료한 이후, K8S 클러스터 구축을 계속한다` +* 순서 : + * 아래의 GUIDE에서, 3-1번을 제외하고 클러스터 구축을 수행한다. + * [K8S MASTER INSTALL GUIDE](https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/K8S_Master#k8s-master-%EC%84%A4%EC%B9%98-%EA%B0%80%EC%9D%B4%EB%93%9C) + + +# B. 별도의 LBNode를 갖지 않고, 클러스터 내에서 HAProxy를 동작시킬 경우 +## 구성 요소 및 버전 +* Keepalived v1.3.5 # MasterNode에 설치 +* HA-Proxy v1.5.18 # MasterNode에 설치 +## Install Steps +0. 변수 설정 및 준비 +1. HAProxy + Keepalived 설치 +2. 설치한 서비스 기동 +3. K8S 클러스터 구축 + +## Step 0. 변수 설정 및 준비 +* 목적 : `HAProxy Node 구축을 위한 변수 설정 및 준비과정` +* 순서 : + * Keepalived 와 HAProxy를 설치 및 동작시키기 위한 변수를 설정한다. + * 클러스터 구성에 사용할 각 Master Node의 IP, VIP에 대한 정보를 입력한다. + ```bash + export MASTER1NAME=master1hostname # 클러스터로 구성할 Master Node의 host명을 각각 입력. + export MASTER2NAME=master2hostname + export MASTER3NAME=master3hostname + + export MASTER1IP=192.168.56.222 # 현재 Node의 IP를 입력. + export MASTER2IP=192.168.56.223 # 다른 Node의 IP를 입력. + export MASTER3IP=192.168.56.224 # 다른 Node의 IP를 입력. + + export MASTERPORT=6443 # 기본적으로 Master Port는 6443을 사용. + export HAPROXYLBPORT=16443 # 반드시 MASTERPORT와 다른 Port를 사용해야 하며, 이경우 Master Join시에 이 변수로 설정한 Port를 사용해야 함. + + export VIP=192.168.56.240 # K8S Master Join시VIP로 사용할 IP를 입력. + ``` + + * Node 구축을 위해 필요한 파일들을 동일한 위치에 다운로드 받는다. + ```bash + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/haproxy_nolb.cfg + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/keepalived_nolb.conf + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/lb_set_script_nolb.sh + wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/master/K8S_Master_LBNode/notify_action.sh + ``` + + * SELinux 관련 플래그를 설정한다. + ```bash + sudo setsebool -P haproxy_connect_any=1 + ``` + + * Node에서 동작 중인 firewalld를 중지 및 비활성화 한다. + ```bash + sudo systemctl stop firewalld && sudo systemctl disable firewalld + ``` + + +## Step.1 HAProxy + Keepalived 설치 +* 목적 : `설치 스크립트를 실행하여 HAProxy와 Keepalived를 MasterNode에 설치` +* 순서 : + * 설치 스크립트에 실행 권한을 주고, 실행한다. + ```bash + sudo chmod +x lb_set_script_nolb.sh + sudo ./lb_set_script_nolb.sh + ``` + + +## Step.2 설치한 서비스 기동 +* 목적 : `HAProxy와 Keepalived 기동` +* 순서 : + * 각 서비스의 설정파일에 Step0 에서 입력한 값들이 올바르게 설정되었는지 확인한다. + ```bash + sudo vi /etc/keepalived/keepalived_nolb.conf + sudo vi /etc/haproxy/haproxy_nolb.cfg + ``` + + * Keepalived 설정파일의 세부내용을 확인/수정한다. + * state 필드는 MASTER or BACKUP을 반드시 수정하며, priority 또한 수정한다. + * interface도 수정해줘야한다. + * unicast_src_ip 는 현재 설치 진행 중인 서버(앞서 설정한 MASTER1IP 변수)이다. + * unicast_peer 는 다른 LB 서버(앞서 설정한 MASTER2IP, MASTER3IP 변수)이다. + ```bash + global_defs { + script_user root root + enable_script_security off + } + + vrrp_script chk_haproxy { + script "/usr/sbin/pidof haproxy" + interval 2 + weight 2 + } + + vrrp_instance VI_1 { + state MASTER # MASTER는 메인 Node, 백업 Node는 BACKUP 으로 설정 + interface enp0s8 # 사용할 interface + virtual_router_id 51 + priority 100 # MASTER의 우선순위를 가장 높게(ex. 100), BACKUP의 경우 그보다 낮게(ex. 99, 98) 설정. + advert_int 1 + authentication { # 인증에 사용될 password(동일하게 맞춰주기만 하면 됨) + auth_type PASS + auth_pass 1111 + } + + unicast_src_ip MASTER1IP # 현재 설치 중인 Node의 local ip + + unicast_peer { + MASTER2IP # 다른 Node의 local ip + MASTER3IP + } + + virtual_ipaddress { + VIP # 클러스터 구성에 사용될 VIP! + } + + notify_master "/bin/sh /etc/keepalived/notify_action.sh MASTER" + notify_backup "/bin/sh /etc/keepalived/notify_action.sh BACKUP" + notify_fault "/bin/sh /etc/keepalived/notify_action.sh FAULT" + notify_stop "/bin/sh /etc/keepalived/notify_action.sh STOP" + + track_script { + chk_haproxy + } + + track_interface { + enp0s8 # 사용할 interface + } + } + ``` + + * HA Proxy 설정파일의 세부내용을 확인/수정한다. + ```bash + global + log 127.0.0.1 local2 + maxconn 2000 + uid 0 + gid 0 + daemon # background process + + defaults + log global # global 설정 사용 + mode tcp # SSL 통신을 위해서는 TCP모드로 (http모드는 SSL 안됨) + option tcplog + option dontlognull # 데이터가 전송되지 않은 연결 로깅 제외 + retries 3 # 연결요청 재시도 횟수 + maxconn 2000 #option redispatch + timeout connect 10s + timeout client 1m + timeout server 1m + + frontend k8s-api + bind 0.0.0.0:HAPROXYLBPORT # Master Node와 동일 Node에 설치시, Master Join을 해당 port로 해야함. + default_backend k8s-api + + backend k8s-api + option tcp-check + balance roundrobin + server MASTER1NAME MASTER1IP:MASTERPORT check # Master 다중화 서버들 정보 기재 + server MASTER2NAME MASTER2IP:MASTERPORT check + server MASTER3NAME MASTER3IP:MASTERPORT check + ``` + + * 각 서비스를 활성화시켜주며 기동하고, 동작을 확인한다. + ```bash + sudo systemctl enable keepalived + sudo systemctl enable haproxy + + sudo systemctl daemon-reload + + sudo systemctl start keepalived + sudo systemctl start haproxy + + sudo systemctl status keepalived + sudo systemctl status haproxy + ``` + +## Step.3 K8S 클러스터 구축 +* 목적 : `이후, K8S 클러스터 구축을 계속한다` +* 순서 : + * 아래의 GUIDE에서, 3-1번을 제외하고 클러스터 구축을 수행한다. + * [K8S MASTER INSTALL GUIDE](https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/K8S_Master#k8s-master-%EC%84%A4%EC%B9%98-%EA%B0%80%EC%9D%B4%EB%93%9C) diff --git a/K8S_Master_LBNode/haproxy.cfg b/K8S_Master_LBNode/haproxy.cfg new file mode 100644 index 00000000..e68587be --- /dev/null +++ b/K8S_Master_LBNode/haproxy.cfg @@ -0,0 +1,28 @@ +global + log 127.0.0.1 local2 + maxconn 2000 + uid 0 + gid 0 + daemon # background process + +defaults + log global # global 설정 사용 + mode tcp # SSL 통신을 위해서는 TCP모드로 (http모드는 SSL 안됨) + option tcplog + option dontlognull # 데이터가 전송되지 않은 연결 로깅 제외 + retries 3 # 연결요청 재시도 횟수 + maxconn 2000 #option redispatch + timeout connect 10s + timeout client 1m + timeout server 1m + +frontend k8s-api + bind 0.0.0.0:HAPROXYLBPORT + default_backend k8s-api + +backend k8s-api + option tcp-check + balance roundrobin + server MASTER1NAME MASTER1IP:MASTERPORT check # Master 다중화 서버들 정보 기재 + server MASTER2NAME MASTER2IP:MASTERPORT check + server MASTER3NAME MASTER3IP:MASTERPORT check diff --git a/K8S_Master_LBNode/haproxy_nolb.cfg b/K8S_Master_LBNode/haproxy_nolb.cfg new file mode 100644 index 00000000..a340f399 --- /dev/null +++ b/K8S_Master_LBNode/haproxy_nolb.cfg @@ -0,0 +1,28 @@ +global + log 127.0.0.1 local2 + maxconn 2000 + uid 0 + gid 0 + daemon # background process + +defaults + log global # global 설정 사용 + mode tcp # SSL 통신을 위해서는 TCP모드로 (http모드는 SSL 안됨) + option tcplog + option dontlognull # 데이터가 전송되지 않은 연결 로깅 제외 + retries 3 # 연결요청 재시도 횟수 + maxconn 2000 #option redispatch + timeout connect 10s + timeout client 1m + timeout server 1m + +frontend k8s-api + bind 0.0.0.0:HAPROXYLBPORT # Master Node와 동일 Node에 설치시, Master Join을 해당 port로 해야함. + default_backend k8s-api + +backend k8s-api + option tcp-check + balance roundrobin + server MASTER1NAME MASTER1IP:MASTERPORT check # Master 다중화 서버들 정보 기재 + server MASTER2NAME MASTER2IP:MASTERPORT check + server MASTER3NAME MASTER3IP:MASTERPORT check diff --git a/K8S_Master_LBNode/keepalived.conf b/K8S_Master_LBNode/keepalived.conf new file mode 100644 index 00000000..6e1e3075 --- /dev/null +++ b/K8S_Master_LBNode/keepalived.conf @@ -0,0 +1,46 @@ +global_defs { + script_user root root + enable_script_security off +} + +vrrp_script chk_haproxy { + script "/usr/sbin/pidof haproxy" + interval 2 + weight 2 +} + +vrrp_instance VI_1 { + state MASTER # MASTER는 메인 LB, 백업 LB는 BACKUP 으로 설정 + interface enp0s8 # 사용할 interface + virtual_router_id 51 + priority 100 # MASTER의 우선순위를 가장 높게(ex. 100), BACKUP의 경우 그보다 낮게(ex. 99, 98) 설정. + advert_int 1 + authentication { # 인증에 사용될 password(동일하게 맞춰주기만 하면 됨) + auth_type PASS + auth_pass 1111 + } + + unicast_src_ip LB1 # LB 서버 local ip + + unicast_peer { + LB2 # 다른 LB 서버 local ip + } + + virtual_ipaddress { + VIP # 클러스터 구성에 사용될 VIP! + } + + notify_master "/bin/sh /etc/keepalived/notify_action.sh MASTER" + notify_backup "/bin/sh /etc/keepalived/notify_action.sh BACKUP" + notify_fault "/bin/sh /etc/keepalived/notify_action.sh FAULT" + notify_stop "/bin/sh /etc/keepalived/notify_action.sh STOP" + + track_script { + chk_haproxy + } + + track_interface { + enp0s8 # 사용할 interface + } + +} diff --git a/K8S_Master_LBNode/keepalived_nolb.conf b/K8S_Master_LBNode/keepalived_nolb.conf new file mode 100644 index 00000000..e64a7d28 --- /dev/null +++ b/K8S_Master_LBNode/keepalived_nolb.conf @@ -0,0 +1,47 @@ +global_defs { + script_user root root + enable_script_security off +} + +vrrp_script chk_haproxy { + script "/usr/sbin/pidof haproxy" + interval 2 + weight 2 +} + +vrrp_instance VI_1 { + state MASTER # MASTER는 메인 LB, 백업 Node는 BACKUP 으로 설정 + interface enp0s8 # 사용할 interface + virtual_router_id 51 + priority 100 # MASTER의 우선순위를 가장 높게(ex. 100), BACKUP의 경우 그보다 낮게(ex. 99, 98) 설정. + advert_int 1 + authentication { # 인증에 사용될 password(동일하게 맞춰주기만 하면 됨) + auth_type PASS + auth_pass 1111 + } + + unicast_src_ip MASTER1IP # 현재 설치 중인 Node의 local ip + + unicast_peer { + MASTER2IP # 다른 Node의 local ip + MASTER3IP + } + + virtual_ipaddress { + VIP # 클러스터 구성에 사용될 VIP! + } + + notify_master "/bin/sh /etc/keepalived/notify_action.sh MASTER" + notify_backup "/bin/sh /etc/keepalived/notify_action.sh BACKUP" + notify_fault "/bin/sh /etc/keepalived/notify_action.sh FAULT" + notify_stop "/bin/sh /etc/keepalived/notify_action.sh STOP" + + track_script { + chk_haproxy + } + + track_interface { + enp0s8 # 사용할 interface + } + +} diff --git a/K8S_Master_LBNode/lb_set_script.sh b/K8S_Master_LBNode/lb_set_script.sh new file mode 100644 index 00000000..d2042952 --- /dev/null +++ b/K8S_Master_LBNode/lb_set_script.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +rpm_pkg_list=("keepalived" "haproxy") + +for pkgname in ${rpm_pkg_list[@]}; +do + echo " " + echo "*** ${pkgname} install... ***" + echo " " + rpm_need_install="sudo yum install -y ${pkgname}" + ${rpm_need_install} +done + +echo " " +echo "*** Finish pkg installation ***" +echo " " +echo "*** File copying and modifying started ***" +echo " " + +SCRIPTPATH=$(dirname `which $0`) + +THISPATH=`echo $SCRIPTPATH` + +sudo cp ${THISPATH}/notify_action.sh /etc/keepalived/notify_action.sh +sudo chmod +x /etc/keepalived/notify_action.sh + +sudo mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf_back +sudo cp -f ${THISPATH}/keepalived.conf /etc/keepalived/keepalived.conf + +sudo sed -i 's/LB1/'"$LB1"'/' /etc/keepalived/keepalived.conf +sudo sed -i 's/LB2/'"$LB2"'/' /etc/keepalived/keepalived.conf + +sudo sed -i 's/VIP/'"$VIP"'/' /etc/keepalived/keepalived.conf + +sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_back +sudo cp -f ${THISPATH}/haproxy.cfg /etc/haproxy/haproxy.cfg + +sudo sed -i 's/MASTER1NAME/'"$MASTER1NAME"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER2NAME/'"$MASTER2NAME"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER3NAME/'"$MASTER3NAME"'/' /etc/haproxy/haproxy.cfg + +sudo sed -i 's/MASTER1IP/'"$MASTER1IP"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER2IP/'"$MASTER2IP"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER3IP/'"$MASTER3IP"'/' /etc/haproxy/haproxy.cfg + +sudo sed -i 's/MASTERPORT/'"$MASTERPORT"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/HAPROXYLBPORT/'"$HAPROXYLBPORT"'/' /etc/haproxy/haproxy.cfg + +echo " " +echo "*** Finish file copying and modifying ***" +echo " " +echo "*** Finish all task in this script ***" +echo " " diff --git a/K8S_Master_LBNode/lb_set_script_nolb.sh b/K8S_Master_LBNode/lb_set_script_nolb.sh new file mode 100644 index 00000000..515e3b37 --- /dev/null +++ b/K8S_Master_LBNode/lb_set_script_nolb.sh @@ -0,0 +1,54 @@ +#!/bin/bash + +rpm_pkg_list=("keepalived" "haproxy") + +for pkgname in ${rpm_pkg_list[@]}; +do + echo " " + echo "*** ${pkgname} install... ***" + echo " " + rpm_need_install="sudo yum install -y ${pkgname}" + ${rpm_need_install} +done + +echo " " +echo "*** Finish pkg installation ***" +echo " " +echo "*** File copying and modifying started ***" +echo " " + +SCRIPTPATH=$(dirname `which $0`) + +THISPATH=`echo $SCRIPTPATH` + +sudo cp ${THISPATH}/notify_action.sh /etc/keepalived/notify_action.sh +sudo chmod +x /etc/keepalived/notify_action.sh + +sudo mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf_back +sudo cp -f ${THISPATH}/keepalived_nolb.conf /etc/keepalived/keepalived.conf + +sudo sed -i 's/MASTER1IP/'"$MASTER1IP"'/' /etc/keepalived/keepalived.conf +sudo sed -i 's/MASTER2IP/'"$MASTER2IP"'/' /etc/keepalived/keepalived.conf +sudo sed -i 's/MASTER3IP/'"$MASTER3IP"'/' /etc/keepalived/keepalived.conf + +sudo sed -i 's/VIP/'"$VIP"'/' /etc/keepalived/keepalived.conf + +sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_back +sudo cp -f ${THISPATH}/haproxy_nolb.cfg /etc/haproxy/haproxy.cfg + +sudo sed -i 's/MASTER1NAME/'"$MASTER1NAME"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER2NAME/'"$MASTER2NAME"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER3NAME/'"$MASTER3NAME"'/' /etc/haproxy/haproxy.cfg + +sudo sed -i 's/MASTER1IP/'"$MASTER1IP"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER2IP/'"$MASTER2IP"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/MASTER3IP/'"$MASTER3IP"'/' /etc/haproxy/haproxy.cfg + +sudo sed -i 's/MASTERPORT/'"$MASTERPORT"'/' /etc/haproxy/haproxy.cfg +sudo sed -i 's/HAPROXYLBPORT/'"$HAPROXYLBPORT"'/' /etc/haproxy/haproxy.cfg + +echo " " +echo "*** Finish file copying and modifying ***" +echo " " +echo "*** Finish all task in this script ***" +echo " " diff --git a/K8S_Master_LBNode/notify_action.sh b/K8S_Master_LBNode/notify_action.sh new file mode 100644 index 00000000..a6a4c3e5 --- /dev/null +++ b/K8S_Master_LBNode/notify_action.sh @@ -0,0 +1,35 @@ +#!/bin/bash +#/etc/keepalived/notify_action.sh +log_file=/var/log/keepalived.log +log_write() +{ + echo "[`date '+%Y-%m-%d %T'`] $1" >> $log_file +} + +[ ! -d /var/keepalived/ ] && mkdir -p /var/keepalived/ + +case "$1" in + "MASTER" ) + echo -n "$1" > /var/keepalived/state + log_write " notify_master" + echo -n "0" > /var/keepalived/vip_check_failed_count + ;; + + "BACKUP" ) + echo -n "$1" > /var/keepalived/state + log_write " notify_backup" + ;; + + "FAULT" ) + echo -n "$1" > /var/keepalived/state + log_write " notify_fault" + ;; + + "STOP" ) + echo -n "$1" > /var/keepalived/state + log_write " notify_stop" + ;; + *) + log_write "notify_action.sh: STATE ERROR!!!" + ;; +esac diff --git a/Kubeflow/README.md b/Kubeflow/README.md index fef20aed..699037d0 100644 --- a/Kubeflow/README.md +++ b/Kubeflow/README.md @@ -29,9 +29,9 @@ * v1.5.1 * https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/Istio 3. Prometheus - * kubeflow의 모니터링 정보를 제공하기 위해 필요하다. - * http://192.168.1.150:10080/hypercloud/hypercloud/wikis/Prometheus-InstallerGuide -4. GPU plug-in (Optional) + * Kubeflow의 모니터링 정보를 제공하기 위해 필요하다. + * https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Prometheus/README.md +4. (Optional) GPU plug-in * Kubernetes cluster 내 node에 GPU가 탑재되어 있으며 AI DevOps 기능을 사용할 때 GPU가 요구될 경우에 필요하다. * https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/Pod_GPU%20plugin @@ -39,23 +39,47 @@ 설치를 진행하기 전 아래의 과정을 통해 필요한 이미지 및 yaml 파일을 준비한다. 1. 이미지 준비 * 아래 링크를 참고하여 폐쇄망에서 사용할 registry를 구축한다. - * 폐쇄망 registry 구축 링크 - * 아래 명령어를 수행하여 Kubeflow 설치 시 필요한 이미지들을 위에서 구축한 registry에 push한다. + * https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Image_Registry/README.md + * 자신이 사용할 registry의 IP와 port를 입력한다. + ```bash + $ export REGISTRY_ADDRESS=1.1.1.1:5000 + ``` + * 아래 명령어를 수행하여 Kubeflow 설치 시 필요한 이미지들을 위에서 구축한 registry에 push하고 이미지들을 tar 파일로 저장한다. tar 파일은 images 디렉토리에 저장된다. ```bash $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/image-push.sh $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/imagelist $ chmod +x ./image-push.sh - $ ./image-push.sh <> + $ ./image-push.sh ${REGISTRY_ADDRESS} + ``` + * 아래 명령어를 수행하여 registry에 이미지들이 잘 push되었는지, 그리고 필요한 이미지들이 tar 파일로 저장되었는지 확인한다. + ```bash + $ curl -X GET ${REGISTRY_ADDRESS}/v2/_catalog + $ ls ./images ``` + * (Optional) 만약 설치에 필요한 이미지들을 pull받아서 tar 파일로 저장하는 작업과 로드하여 push하는 작업을 따로 수행하고자 한다면 image-push.sh이 아니라 image-save.sh, image-load.sh를 각각 실행하면 된다. + * image-save.sh을 실행하면 설치에 필요한 이미지들을 pull 받아서 images 디렉토리에 tar 파일로 저장한다. + ```bash + $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/image-save.sh + $ chmod +x ./image-save.sh + $ ./image-save.sh + $ ls ./images + ``` + * 위에서 저장한 images 디렉토리와 image-load.sh을 폐쇄망 환경으로 옮긴 후 실행하면 폐쇄망 내 구축한 registry에 이미지들을 push할 수 있다. image-load.sh은 images 디렉토리와 같은 경로에서 실행해야만 한다. + ```bash + $ chmod +x ./image-load.sh + $ ./image-load.sh ${REGISTRY_ADDRESS} + $ curl -X GET ${REGISTRY_ADDRESS}/v2/_catalog + ``` 2. Yaml 파일 및 script 파일 준비 * 아래 명령어를 수행하여 Kubeflow 설치에 필요한 yaml 파일들과 script 파일들을 다운로드 받는다. ```bash $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/sed.sh - $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/kustomize.tar.gz - $ wget https://github.com/kubeflow/kfctl/releases/download/v1.0.2/kfctl_v1.0.2-0-ga476281_linux.tar.gz + $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/kustomize_local.tar.gz + $ wget https://raw.githubusercontent.com/tmax-cloud/hypercloud-install-guide/4.1/Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2_local.yaml + $ wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0-0-gbc038f9_linux.tar.gz ``` 3. 앞으로의 진행 - * Step 0 ~ 4 중 Step 0, 2 비고를 참고하여 진행한다. 나머지는 그대로 진행하면 된다. + * Step 0 ~ 4 중 Step 0, 2, 3은 비고를 참고하여 진행한다. 나머지는 그대로 진행하면 된다. ## Install Steps 0. [kfctl 설치](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Kubeflow/README.md#step-0-kfctl-%EC%84%A4%EC%B9%98) @@ -68,12 +92,12 @@ * 목적 : `Kubeflow component를 배포 및 관리하기 위한 커맨드 라인툴인 kfctl을 설치한다.` * 생성 순서 : 아래 명령어를 수행하여 kfctl을 설치한다. (Kubeflow v1.0.2 기준) ```bash - $ wget https://github.com/kubeflow/kfctl/releases/download/v1.0.2/kfctl_v1.0.2-0-ga476281_linux.tar.gz - $ tar xzvf kfctl_v1.0.2-0-ga476281_linux.tar.gz + $ wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0-0-gbc038f9_linux.tar.gz + $ tar xzvf kfctl_v1.2.0-0-gbc038f9_linux.tar.gz $ sudo mv kfctl /usr/bin ``` * 비고 : - * 폐쇄망 환경일 경우 첫 번째 명령어로 github에 있는 kfctl을 다운로드받는 대신 미리 준비한 kfctl을 다운받으면 된다. + * 폐쇄망 환경일 경우 kfctl_v1.2.0-0-gbc038f9_linux.tar.gz을 github에서 받는 것이 아니라 미리 다운로드 해둔 것을 사용하면 된다. ## Step 1. 설치 디렉토리 생성 * 목적 : `Kubeflow의 설치 yaml이 저장될 설치 디렉토리를 생성하고 해당 경로로 이동한다.` @@ -98,12 +122,12 @@ ``` * 정상적으로 완료되면 kustomize라는 디렉토리가 생성된다. * 비고 : - * 폐쇄망 환경일 경우 설치 디렉토리에 미리 다운로드받은 sed.sh, kustomize.tar.gz 파일을 옮긴다. + * 폐쇄망 환경일 경우 설치 디렉토리 ${KF_DIR}에 미리 다운로드받은 sed.sh, kustomize_local.tar.gz 파일을 옮긴다. * 아래 명령어를 통해 Kustomize 리소스의 압축을 풀고 yaml 파일들에서 이미지들을 pull 받을 registry를 바꿔준다. ```bash - $ tar xvfz kustomize.tar.gz + $ tar xvfz kustomize_local.tar.gz $ chmod +x ./sed.sh - $ ./sed.sh <> ${KF_DIR}/kustomize + $ ./sed.sh ${REGISTRY_ADDRESS} ${KF_DIR}/kustomize ``` ## Step 3. Kubeflow 배포 @@ -119,6 +143,12 @@ ![pasted image 0](https://user-images.githubusercontent.com/63379907/90479302-6aedb380-e169-11ea-8c6c-9c1b4e15517a.png) * 설치에는 약 10분 정도가 소요된다. * 비고 : + * 폐쇄망 환경일 경우 설치 디렉토리 ${KF_DIR}에 미리 다운로드받은 kfctl_hypercloud_kubeflow.v1.0.2_local.yaml 파일을 옮긴다. + * 아래 명령어를 수행하여 Kubeflow를 배포한다. + ```bash + $ export CONFIG_FILE=${KF_DIR}/kfctl_hypercloud_kubeflow.v1.0.2_local.yaml + $ kfctl apply -V -f ${CONFIG_FILE} + ``` * 기존 Kubeflow에서 수정된 점 * Istio 1.5.1 호환을 위해 KFServing의 controller 수정 * Workflow template을 사용하기 위한 argo controller 버전 업 @@ -168,3 +198,16 @@ $ echo '{"apiVersion":"security.istio.io/v1beta1","kind":"PeerAuthentication","metadata":{"annotations":{},"name":"default","namespace":"istio-system"},"spec":{"mtls":{"mode":"DISABLE"}}}' |cat > disable-mtls.json $ kubectl apply -f disable-mtls.json ``` +## 기타 : kubeflow 삭제 +* 목적 : `kubeflow 설치 시에 배포된 모든 리소스를 삭제 한다.` +* 생성 순서 : + * 아래 명령어를 수행하여 kubeflow 모듈을 삭제한다. + ```bash + $ export CONFIG_URI="https://raw.githubusercontent.com/tmax-cloud/kubeflow-manifests/kubeflow-manifests-v1.0.2/kfctl_hypercloud_kubeflow.v1.0.2.yaml" + $ kfctl delete -V -f ${CONFIG_URI} + ``` +* 비고 : + * kfctl 1.1버전 이상부터 리소스의 삭제가 정상적으로 이루어진다. kfctl 버전은 다음명령어를 통해 확인할 수 있다. + ```bash + $ kfctl version + ``` diff --git a/Kubeflow/demo/1.notebook_closednw.yaml b/Kubeflow/demo/1.notebook_closednw.yaml new file mode 100644 index 00000000..db50b3c1 --- /dev/null +++ b/Kubeflow/demo/1.notebook_closednw.yaml @@ -0,0 +1,42 @@ +apiVersion: hyperflow.tmax.io/v1 +kind: Notebook +metadata: + labels: + app: demo-notebook + name: demo-notebook + namespace: demo +spec: + template: + spec: + containers: + - env: + ## 폐쇄망 registry 주소 (수정 필요) + - name: REGISTRY + value: "192.168.6.181:5000" + ## notebook image 정의 (수정 필요) + image: '192.168.6.181:5000/tmaxcloudck/kubeflow-jupyter-lab:v0.1' + name: demo + resources: + ## notebook 사양 정의 + requests: + cpu: "0.5" + memory: 1.0Gi + volumeMounts: + - mountPath: /home/jovyan + name: demo-pvc + - mountPath: /dev/shm + name: dshm + serviceAccountName: default-editor + ttlSecondsAfterFinished: 300 + ## 사용할 volume 정의 + volumes: + - name: demo-pvc + persistentVolumeClaim: + claimName: demo-pvc + - emptyDir: + medium: Memory + name: dshm + ## 사용할 volumeClaim 정의 : 없으면 새로 pvc를 생성 + volumeClaim: + - name: demo-pvc + size: 10Gi \ No newline at end of file diff --git a/Kubeflow/demo/7.wf-template.yaml b/Kubeflow/demo/7.wf-template.yaml new file mode 100644 index 00000000..05f263a6 --- /dev/null +++ b/Kubeflow/demo/7.wf-template.yaml @@ -0,0 +1,206 @@ +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: demo-workflowtemplate + namespace: demo +spec: + ## 공통 arg 정의 + arguments: + parameters: + - name: pvc_name + value: demo-pvc + ## 시작 task(template) 설정 + entrypoint: step + serviceAccountName: default-editor + templates: + ## 수행 flow를 정의 + - name: step + steps: + - - name: tunning + template: katib + - - name: training + template: tfjob + arguments: + parameters: + - name: tfjob-arg-lrrate + value: '{{steps.tunning.outputs.parameters.learning_rate}}' + - name: tfjob-arg-dorate + value: '{{steps.tunning.outputs.parameters.dropout_rate}}' + - - name: serving + template: kfserving + + ## Katib를 수행하는 task를 정의 + - name: katib + metadata: + annotations: + sidecar.istio.io/inject: "false" + resource: + action: create + successCondition: status.trialsSucceeded==5 + manifest: | + apiVersion: "kubeflow.org/v1alpha3" + kind: Experiment + metadata: + namespace: demo + labels: + controller-tools.k8s.io: "1.0" + name: demo-experiment + spec: + ## 목표 정의 : validation-accuracy를 maxmize + objective: + type: maximize + goal: 0.99 + objectiveMetricName: Validation-accuracy + additionalMetricNames: + - accuracy + - loss + - Validation-loss + ## 메트릭 수집 방식 정의 : /result/mnist.log의 File을 통해 수집 + metricsCollectorSpec: + source: + fileSystemPath: + path: "/result/mnist.log" + kind: File + collector: + kind: File + ## hyperParameter 탐색 algorithm 정의 + algorithm: + algorithmName: random + ## 분산 학습 설정 : 최대 5 학습, 동시에 3 학습씩 진행 + parallelTrialCount: 3 + maxTrialCount: 5 + maxFailedTrialCount: 3 + ## 살펴볼 hyperParameter들과 그 범위 정의 : learning_rate와 dropout_rate를 살펴봄 + parameters: + - name: --learning_rate + parameterType: double + feasibleSpace: + min: "0.01" + max: "0.03" + - name: --dropout_rate + parameterType: double + feasibleSpace: + min: "0.1" + max: "0.9" + ## ML Model을 학습시킬 Job 정의 + trialTemplate: + goTemplate: + rawTemplate: |- + apiVersion: batch/v1 + kind: Job + metadata: + name: {{.Trial}} + namespace: {{.NameSpace}} + spec: + template: + spec: + containers: + - name: {{.Trial}} + image: docker.io/rhojw/sample-job:3C8CE2EE + command: + - "python" + - "/app/fmnist-save-model-renew.py" + {{- with .HyperParameters}} + {{- range .}} + - "{{.Name}}={{.Value}}" + {{- end}} + {{- end}} + resources: + limits: + nvidia.com/gpu: 1 + restartPolicy: Never + outputs: + parameters: + - name: learning_rate + valueFrom: + jsonPath: "{.status.currentOptimalTrial.parameterAssignments[?(@.name=='--learning_rate')].value}" + - name: dropout_rate + valueFrom: + jsonPath: "{.status.currentOptimalTrial.parameterAssignments[?(@.name=='--dropout_rate')].value}" + + ## TFJob을 수행하는 task를 정의 + - name: tfjob + metadata: + annotations: + sidecar.istio.io/inject: "false" + inputs: + parameters: + - name: tfjob-arg-lrrate + - name: tfjob-arg-dorate + resource: + action: create + successCondition: status.replicaStatuses.Worker.succeeded==1 + manifest: | + apiVersion: kubeflow.org/v1 + kind: TFJob + metadata: + name: demo-tfjob + namespace: demo + spec: + tfReplicaSpecs: + Worker: + replicas: 1 + template: + metadata: + annotations: + sidecar.istio.io/inject: "false" + name: fairing-deployer + spec: + ## hyperparameter값을 설정 : learningRate, dropoutRate + containers: + - command: + - python + - /app/fmnist-save-model-renew.py + - "--learning_rate={{inputs.parameters.tfjob-arg-lrrate}}" + - "--dropout_rate={{inputs.parameters.tfjob-arg-dorate}}" + env: + - name: FAIRING_RUNTIME + value: "1" + image: rhojw/sample-job:3C8CE2EE + name: tensorflow + resources: + limits: + cpu: 1 + memory: 1.86Gi + securityContext: + runAsUser: 0 + volumeMounts: + - mountPath: /result + name: fairing-volume-demo-pvc + workingDir: /app/ + restartPolicy: Never + ## Model이 저장될 storage 정의 + volumes: + - name: fairing-volume-demo-pvc + persistentVolumeClaim: + claimName: '{{workflow.parameters.pvc_name}}' + + ## KFServing을 수행하는 task를 정의 + - name: kfserving + metadata: + annotations: + sidecar.istio.io/inject: "false" + resource: + action: create + successCondition: status.traffic==100 + manifest: | + apiVersion: serving.kubeflow.org/v1alpha2 + kind: InferenceService + metadata: + name: demo-inferenceservice + namespace: demo + spec: + default: + predictor: + ## inference server 정의 + tensorflow: + resources: + limits: + cpu: 100m + memory: 1Gi + requests: + cpu: 100m + memory: 1Gi + runtimeVersion: 1.14.0 + ## Model 위치 설정 + storageUri: pvc://{{workflow.parameters.pvc_name}}/saved_model \ No newline at end of file diff --git a/Kubeflow/demo/7.workflow.yaml b/Kubeflow/demo/7.workflow.yaml new file mode 100644 index 00000000..1d6596e9 --- /dev/null +++ b/Kubeflow/demo/7.workflow.yaml @@ -0,0 +1,8 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Workflow +metadata: + name: demo-workflow + namespace: demo +spec: + workflowTemplateRef: + name: demo-workflowtemplate \ No newline at end of file diff --git a/Kubeflow/demo/Dockerfile b/Kubeflow/demo/Dockerfile new file mode 100644 index 00000000..efd71a81 --- /dev/null +++ b/Kubeflow/demo/Dockerfile @@ -0,0 +1,153 @@ +# Copyright (c) Jupyter Development Team. +# Distributed under the terms of the Modified BSD License. + +ARG BASE_IMAGE=tensorflow/tensorflow:2.1.0-py3-jupyter + +FROM $BASE_IMAGE + +ARG TF_SERVING_VERSION=0.0.0 +ARG NB_USER=jovyan + +# TODO: User should be refactored instead of hard coded jovyan + +USER root + +ENV DEBIAN_FRONTEND noninteractive + +ENV NB_USER $NB_USER + +ENV NB_UID 1000 +ENV HOME /home/$NB_USER +ENV NB_PREFIX / + + +# Use bash instead of sh +SHELL ["/bin/bash", "-c"] + +RUN apt-get update && apt-get install -yq --no-install-recommends \ + apt-transport-https \ + build-essential \ + bzip2 \ + ca-certificates \ + curl \ + g++ \ + git \ + gnupg \ + graphviz \ + locales \ + lsb-release \ + openssh-client \ + sudo \ + unzip \ + vim \ + wget \ + zip \ + emacs \ + python3-pip \ + python3-dev \ + python3-setuptools \ + npm \ + && apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +# Install Nodejs for jupyterlab-manager +RUN curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - +RUN apt-get update && apt-get install -yq --no-install-recommends \ + nodejs \ + && apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +ENV DOCKER_CREDENTIAL_GCR_VERSION=1.4.3 +RUN curl -LO https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${DOCKER_CREDENTIAL_GCR_VERSION}/docker-credential-gcr_linux_amd64-${DOCKER_CREDENTIAL_GCR_VERSION}.tar.gz && \ + tar -zxvf docker-credential-gcr_linux_amd64-${DOCKER_CREDENTIAL_GCR_VERSION}.tar.gz && \ + mv docker-credential-gcr /usr/local/bin/docker-credential-gcr && \ + rm docker-credential-gcr_linux_amd64-${DOCKER_CREDENTIAL_GCR_VERSION}.tar.gz && \ + chmod +x /usr/local/bin/docker-credential-gcr + +# Install AWS CLI +RUN curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "/tmp/awscli-bundle.zip" && \ + unzip /tmp/awscli-bundle.zip && ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws && \ + rm -rf ./awscli-bundle + +# Install Azure CLI +RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null && \ + AZ_REPO=$(lsb_release -cs) && \ + echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | tee /etc/apt/sources.list.d/azure-cli.list && \ + apt-get update && \ + apt-get install azure-cli + + +RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \ + locale-gen + +ENV LC_ALL en_US.UTF-8 +ENV LANG en_US.UTF-8 +ENV LANGUAGE en_US.UTF-8 + +# Create NB_USER user with UID=1000 and in the 'users' group +# but allow for non-initial launches of the notebook to have +# $HOME provided by the contents of a PV +RUN useradd -M -s /bin/bash -N -u $NB_UID $NB_USER && \ + chown -R ${NB_USER}:users /usr/local/bin && \ + mkdir -p $HOME + +RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \ + echo "deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" > /etc/apt/sources.list.d/google-cloud-sdk.list && \ + curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \ + apt-get update && \ + apt-get install -y google-cloud-sdk kubectl + +# Install Tini - used as entrypoint for container +RUN cd /tmp && \ + wget --quiet https://github.com/krallin/tini/releases/download/v0.18.0/tini && \ + echo "12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855 *tini" | sha256sum -c - && \ + mv tini /usr/local/bin/tini && \ + chmod +x /usr/local/bin/tini + +# Install docker +RUN apt-get install software-properties-common -y && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \ + apt-key add - && \ + add-apt-repository \ + "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \ + apt-get update && \ + apt-cache policy docker-ce && \ + apt-get install -y docker-ce + +# NOTE: Beyond this point be careful of breaking out +# or otherwise adding new layers with RUN, chown, etc. +# The image size can grow significantly. + +# Install base python3 packages +RUN pip3 --no-cache-dir install \ + jupyter-console==6.0.0 \ + jupyterlab \ + xgboost \ + kubeflow-fairing==1.0.1 \ + pandas \ + numpy \ + matplotlib \ + sklearn \ + google-auth==1.21.1 \ + six==1.15 \ + urllib3==1.25.4 \ + msrestazure \ + easydict + +RUN pip3 uninstall -y enum34 +#RUN pip3 install urllib3==1.25.4 + +RUN rm -rf /usr/local/lib/python3.6/dist-packages/kubeflow/fairing/constants/constants.py +ADD ./constants.py /usr/local/lib/python3.6/dist-packages/kubeflow/fairing/constants +ADD ./setup.py /usr/local/lib/python3.6/dist-packages +ADD ./requirements.txt /usr/local/lib/python3.6/dist-packages +WORKDIR /usr/local/lib/python3.6/dist-packages +RUN python3 setup.py install + +RUN docker-credential-gcr configure-docker && chown ${NB_USER}:users $HOME/.docker/config.json + +# Configure container startup +EXPOSE 8888 +#USER jovyan +ENTRYPOINT ["tini", "--"] +CMD ["sh","-c", "jupyter lab --notebook-dir=/home/${NB_USER} --ip=0.0.0.0 --no-browser --allow-root --port=8888 --NotebookApp.token='' --NotebookApp.password='' --NotebookApp.allow_origin='*' --NotebookApp.base_url=${NB_PREFIX}"] diff --git a/Kubeflow/demo/KFServing-fairing.ipynb b/Kubeflow/demo/KFServing-fairing.ipynb new file mode 100644 index 00000000..3e6b6de6 --- /dev/null +++ b/Kubeflow/demo/KFServing-fairing.ipynb @@ -0,0 +1,165 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[W 200415 10:00:25 append:50] Building image using Append builder...\n", + "[I 200415 10:00:25 base:105] Creating docker context: /tmp/fairing_context_e00hqtan\n", + "[I 200415 10:00:25 converted_notebook:127] Converting KFServing-fairing.ipynb to KFServing-fairing.py\n", + "[I 200415 10:00:25 docker_creds_:234] Loading Docker credentials for repository 'brightfly/kubeflow-kfserving:latest'\n", + "[W 200415 10:00:28 append:54] Image successfully built in 2.7561585219809785s.\n", + "[W 200415 10:00:28 append:94] Pushing image rhojw/kfserving:6962115B...\n", + "[I 200415 10:00:28 docker_creds_:234] Loading Docker credentials for repository 'rhojw/kfserving:6962115B'\n", + "[W 200415 10:00:28 append:81] Uploading rhojw/kfserving:6962115B\n", + "[I 200415 10:00:29 docker_session_:280] Layer sha256:fcb6b3d82de6c9870be99da3b3aba1ee9ae6fd451f58a349297c90d5a12620e1 exists, skipping\n", + "[I 200415 10:00:29 docker_session_:280] Layer sha256:d391d47713783ed4de6f856d63d4192240e9c93d782b158284aedc307cbe4fb6 exists, skipping\n", + "[I 200415 10:00:29 docker_session_:280] Layer sha256:8f94a3a4572995fbbfcb21a5ad2796af7329e8d90a8bf97f615e14a99d848213 exists, skipping\n", + "[I 200415 10:00:29 docker_session_:280] Layer sha256:772bb01b5acf1c08f32de033c32c9f807d047164867a11ffdc96669762b29725 exists, skipping\n", + "[I 200415 10:00:29 docker_session_:280] Layer sha256:ca7e1bbea389949863f72a88b675dd8f888f40a79be6573c1e2f0095335e68ab exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:59b774443140f464249d4af7a526ecee8c2476bad1e5c5c15d74707b5b2b7227 exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:04de2638d2c074bedb4d1195ea9894e67a647b0f076702afce5a9e6f811b0f40 exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:43bf3e3107f525b7dea5b3cdfd15f62666066202105a0aaddbab2a02aefad1f7 exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:dcfa0aa1ae2c54d0de4b6ad0ee4d9f795c93368ae7d87801784c1f8ae624ac33 exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:d918eaefd9de8a1595f56900fdb8ab65a03d64b1da1078da7c0b0bf6f7552a14 exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:5067aa3d4931716bd5b3473f05f31513fa4a124c03db0a51f6d7baf2a16f700b exists, skipping\n", + "[I 200415 10:00:30 docker_session_:280] Layer sha256:a667b2dd2f883977fa9a4eb368637cb2e6dc8d1d59407c6a00188a3df8b62e6f exists, skipping\n", + "[I 200415 10:00:31 docker_session_:280] Layer sha256:c513d77460ba16f23ee2eb25f8a99c95b5bd67959caa8d1a95d241c793a0ed05 exists, skipping\n", + "[I 200415 10:00:31 docker_session_:280] Layer sha256:8f0fdd3eaac017684c880d836abdaf02f9db7ac8ecca970356482e3d8e315650 exists, skipping\n", + "[I 200415 10:00:31 docker_session_:280] Layer sha256:87b2c33457cb85bf8ea16ce26c1496cdb8d2627237b551c3eeb90a9aa8fd0aed exists, skipping\n", + "[I 200415 10:00:31 docker_session_:280] Layer sha256:5bf324b442328ad6fb1825b37923ad8850e8bc148e9cbfbd0971f951f5f12a81 exists, skipping\n", + "[I 200415 10:00:31 docker_session_:280] Layer sha256:27622921edb2aae9e0cb64f71a81b4a7cfef8f6a6c766514a018815834ba4e14 exists, skipping\n", + "[I 200415 10:00:32 docker_session_:280] Layer sha256:a2d410a0f39cff43a6ad7c800424f6c9dad59813cbde66b65f229e04d6655775 exists, skipping\n", + "[I 200415 10:00:32 docker_session_:284] Layer sha256:519efdf70858769a3ff09fbd3f79b17c82e981afd718f864d69df04afbfa2298 pushed.\n", + "[I 200415 10:00:34 docker_session_:284] Layer sha256:aa3f7829fe208e7d6acac15eda13c119e7f91fba51d23a44a32c366251f7c69a pushed.\n", + "[I 200415 10:00:34 docker_session_:334] Finished upload of: rhojw/kfserving:6962115B\n", + "[W 200415 10:00:34 append:99] Pushed image rhojw/kfserving:6962115B in 6.468625173991313s.\n" + ] + } + ], + "source": [ + "from kubernetes import client\n", + "from kfserving import KFServingClient\n", + "from kfserving import constants\n", + "from kfserving import utils\n", + "from kfserving import V1alpha2EndpointSpec\n", + "from kfserving import V1alpha2PredictorSpec\n", + "from kfserving import V1alpha2TensorflowSpec\n", + "from kfserving import V1alpha2InferenceServiceSpec\n", + "from kfserving import V1alpha2InferenceService\n", + "from kubernetes.client import V1ResourceRequirements\n", + "import os\n", + "import sys\n", + "import argparse\n", + "import logging\n", + "import time\n", + "\n", + "## kfserving module을 이용해 python code로 직접 클러스터에 serving을 하는 과정\n", + "class KFServing(object):\n", + " def run(self):\n", + " parser = argparse.ArgumentParser()\n", + " parser.add_argument('--namespace', required=False, default='kubeflow')\n", + " parser.add_argument('--storage_uri', required=False, default='/mnt/export')\n", + " parser.add_argument('--name', required=False, default='kfserving-sample') \n", + " args = parser.parse_args()\n", + " namespace = args.namespace\n", + " serving_name = args.name\n", + " api_version = constants.KFSERVING_GROUP + '/' + constants.KFSERVING_VERSION\n", + " \n", + " ## inference server spec 정의\n", + " default_endpoint_spec = V1alpha2EndpointSpec(\n", + " predictor=V1alpha2PredictorSpec(\n", + " tensorflow=V1alpha2TensorflowSpec(\n", + " storage_uri=args.storage_uri,\n", + " resources=V1ResourceRequirements(\n", + " requests={'cpu':'100m','memory':'1Gi'},\n", + " limits={'cpu':'100m', 'memory':'1Gi'}))))\n", + " ## kfserving crd(inferenceservice) 정의\n", + " isvc = V1alpha2InferenceService(api_version=api_version,\n", + " metadata=client.V1ObjectMeta(\n", + " name=serving_name, namespace=namespace),\n", + " spec=V1alpha2InferenceServiceSpec(default=default_endpoint_spec)) \n", + " \n", + " KFServing = KFServingClient()\n", + " KFServing.create(isvc)\n", + " print('waiting 5 sec for Creating InferenceService')\n", + " time.sleep(5)\n", + " \n", + " KFServing.get(serving_name, namespace=namespace, watch=True, timeout_seconds=300)\n", + "\n", + "## serving하는 python 코드를 fairing으로 imaging 하는 작업\n", + "if __name__ == '__main__':\n", + " if os.getenv('FAIRING_RUNTIME', None) is None:\n", + " from kubeflow.fairing.builders.append.append import AppendBuilder\n", + " from kubeflow.fairing.preprocessors.converted_notebook import \\\n", + " ConvertNotebookPreprocessor\n", + "\n", + " DOCKER_REGISTRY = 'rhojw'\n", + " base_image = 'brightfly/kubeflow-kfserving:latest'\n", + " image_name = 'kfserving'\n", + "\n", + " builder = AppendBuilder(\n", + " registry=DOCKER_REGISTRY,\n", + " image_name=image_name,\n", + " base_image=base_image,\n", + " push=True,\n", + " preprocessor=ConvertNotebookPreprocessor(\n", + " notebook_file=\"KFServing-fairing.ipynb\"\n", + " )\n", + " )\n", + " builder.build()\n", + " else:\n", + " serving = KFServing()\n", + " serving.run()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "kubeflow_notebook": { + "autosnapshot": false, + "docker_image": "brightfly/kubeflow-jupyter-lab:tf2.0-gpu", + "experiment": { + "id": "", + "name": "" + }, + "experiment_name": "", + "katib_run": false, + "pipeline_description": "", + "pipeline_name": "", + "snapshot_volumes": false, + "steps_defaults": [], + "volumes": [] + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.8" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/Kubeflow/demo/readMe.md b/Kubeflow/demo/readMe.md index e38a0291..17a2c078 100644 --- a/Kubeflow/demo/readMe.md +++ b/Kubeflow/demo/readMe.md @@ -1,5 +1,5 @@ # Hyperflow를 사용한 AI 개발 시나리오 가이드 -가이드 문의 CK2-4팀 +가이드 문의 CK1-4팀 주의 사항 : - notebook-controller-go image b0.0.2 이상 버전, hypercloud-console image 4.1.2.3.0 이상 버전에서 notebook이 UI에 정상 표기 @@ -55,9 +55,12 @@ Fashion-MNIST 데이터를 활용하여 Image가 어떤 Fashion Item인지 추 ![1.notebook.PNG](./img/1.notebook.PNG) - 참고 : [1.notebook.yaml](./1.notebook.yaml) + - 폐쇄망 환경의 경우 : [1.notebook_closednw.yaml](./1.notebook_closednw.yaml) *시나리오에서는 여러 커스텀 패키지가 포함된 custom jupyterNotebook image를 사용하였다. (brightfly/kubeflow-jupyter-lab:tf2.0-gpu) +*폐쇄망 환경의 경우 시나리오에서 사용되는 파이썬 패키지가 포함되어 있고 fairing 코드가 수정된 custom jupyterNotebook image를 사용하였다. (tmaxcloudck/kubeflow-jupyter-lab:v0.1) + - 정상적인 배포를 확인하기 위해, action->connect 버튼을 눌러 jupyter진입을 확인하자. ![1.notebook-connect.PNG](./img/1.notebook-connect.PNG) @@ -65,8 +68,14 @@ Fashion-MNIST 데이터를 활용하여 Image가 어떤 Fashion Item인지 추 ## Step 2. ML model을 코딩하고, 클라우드 작업을 위한 image 생성하기 - tensorflow 모듈을 활용하여 ML 코드를 작성하고, kubeflow 모듈을 활용하여 ML image를 배포한다. - - 정상적으로 image를 배포하기 위해, jupyterNotebook container에 docker registry 인증정보를 넣어야한다. - - 시나리오에서는 public registry인 docker hub를 활용하였고, 인증이 적용된 private registry 또한 사용 가능하다. + - 정상적으로 image를 배포하기 다음 두가지 선행 작업이 필요하다. + - pod이 떠있는 node에 docker를 설치한다. (kubernetes container-runtime이 crio라면 설치되어있지 않은 경우가 있음) + - jupyterNotebook container에 docker registry 인증정보를 넣어야 한다. + - 시나리오에서는 public registry인 docker hub를 활용하였고, 인증이 적용된 private registry 또한 사용 가능하다. + - 폐쇄망 환경일 경우 아래 인증 방법을 수행할 필요 없이 아래 명령어만 수행한다. + ```bash + $ kubectl -n demo create configmap docker-config + ``` ### 인증 방법 1) 로컬 개발 환경의 docker registry 인증정보 사용 - docker에 로그인 되어있는 로컬 개발환경에서 config.json을 복사하여, jupyterNotebook container에 붙여넣는다. @@ -99,7 +108,9 @@ ls home/jovyan/.docker/ cp $HOME/.docker/config.json /home/jovyan/.docker/config.json ``` - - 위의 작업이 끝났다면, code run을 하여 이미지를 배포하자. (UI는 jupyter 버전에 따라 다를 수 있음) + - 위의 작업이 끝났다면, 코드 내 DOCKER_REGISTRY를 자신이 사용할 registry로 변경한다. + - 폐쇄망 환경의 경우 fairing에서 사용할 base_image 또한 폐쇄망 내 registry에서 받아오도록 변경한다. + - Run을 하여 이미지를 배포하자. (UI는 jupyter 버전에 따라 다를 수 있음) ![2.fmnist-save-model-renew.PNG](./img/2.fmnist-save-model-renew.PNG) - 참고 : [fmnist-save-model-renew.ipynb](./fmnist-save-model-renew.ipynb) @@ -109,9 +120,9 @@ cp $HOME/.docker/config.json /home/jovyan/.docker/config.json pip install kubeflow-fairing --upgrade ``` -*실행이 잘 되지 않는다면, pythonNotebook의 kerner을 리셋 후 다시 code run을 진행하자. (code run 옆에 커널새로고침 버튼 클릭) +*실행이 잘 되지 않는다면, pythonNotebook의 kernel을 리셋 후 다시 code run을 진행하자. (code run 옆에 커널 새로고침 버튼 클릭) - - 아래와 같이 docker hub에 rhojw/sample-job:3C8CE2EE 의 image가 배포된 것을 확인할 수 있다. 이후 Step에 사용할 image이다. + - 아래와 같이 docker hub에 rhojw/sample-job:3C8CE2EE 의 image가 배포된 것을 확인할 수 있다. 이후 Step에서 사용할 image이다. ![2.docker-image.PNG](./img/2.docker-image.PNG) @@ -204,6 +215,6 @@ kubectl get service -n istio-system kfserving-ingressgateway ![5.workflow.PNG](./img/5.workflow.PNG) - - 참고 : [5.workflow.yaml](5.workflow.yaml) + - 참고 : [6.workflow.yaml](6.workflow.yaml) -*serving 같은 경우 kfp python module을 사용하여 image로 만들고, 이를 workflow task로 등록하여 생성하였다. +*serving 같은 경우 kfp python module을 사용하여 image로 만들고, 이를 workflow task로 등록하여 생성하였다. 참고 : [KFServing-fairing.ipynb](KFServing-fairing.ipynb) diff --git a/Kubeflow/image-load.sh b/Kubeflow/image-load.sh new file mode 100644 index 00000000..e4c7fdbc --- /dev/null +++ b/Kubeflow/image-load.sh @@ -0,0 +1,27 @@ +#!/bin/bash + +registry="" + +if [ $# -eq 1 ]; then + registry=$1 +else + echo "[$0] ERROR!! Invalid argument count" + echo "[$0] [Usage] $0 192.168.6.110:5000" + exit 1 +fi + +image_num=$(cat imagelist | wc -l) +echo "[$0] Load ${image_num} images & Push to ${registry}" + +i=1 +cat imagelist | while read line +do + echo "[$0] [ ${i} / ${image_num} ] $line" + name=`echo $line |tr '/' '-'` + sudo docker load < ./images/${name}.tar + sudo docker tag $line ${registry}/$line + sudo docker push ${registry}/$line + let i+=1 +done + +echo "[$0] Done" diff --git a/Kubeflow/image-push.sh b/Kubeflow/image-push.sh index 14998989..acc63259 100644 --- a/Kubeflow/image-push.sh +++ b/Kubeflow/image-push.sh @@ -11,16 +11,20 @@ else fi image_num=$(cat imagelist | wc -l) -echo "[$0] Pull ${image_num} images & Push to ${registry}" +echo "[$0] Pull ${image_num} images & Save as tar files & Push to ${registry}" + +mkdir ./images i=1 cat imagelist | while read line do echo "[$0] [ ${i} / ${image_num} ] $line" - let i+=1 sudo docker pull $line sudo docker tag $line ${registry}/$line + name=`echo $line |tr '/' '-'` + sudo docker save $line > ./images/${name}.tar sudo docker push ${registry}/$line + let i+=1 done echo "[$0] Done" diff --git a/Kubeflow/image-save.sh b/Kubeflow/image-save.sh new file mode 100644 index 00000000..6555cc6d --- /dev/null +++ b/Kubeflow/image-save.sh @@ -0,0 +1,18 @@ +#!/bin/bash + +image_num=$(cat imagelist | wc -l) +echo "[$0] Pull ${image_num} images & Save as tar files" + +mkdir ./images + +i=1 +cat imagelist | while read line +do + echo "[$0] [ ${i} / ${image_num} ] $line" + sudo docker pull $line + name=`echo $line |tr '/' '-'` + sudo docker save $line > ./images/${name}.tar + let i+=1 +done + +echo "[$0] Done" diff --git a/Kubeflow/imagelist b/Kubeflow/imagelist index 157d44e9..afe189f8 100644 --- a/Kubeflow/imagelist +++ b/Kubeflow/imagelist @@ -29,25 +29,27 @@ gcr.io/kubeflow-images-public/katib/v1alpha3/katib-db-manager:v0.8.0 mysql:8 gcr.io/kubeflow-images-public/katib/v1alpha3/katib-ui:v0.8.0 docker.io/istio/proxyv2:1.5.1 -gcr.io/kfserving/batcher:v0.4.0 +kfserving/batcher:v0.4.0 gcr.io/kfserving/alibi-explainer -gcr.io/kfserving/logger:0.4.0 -tensorflow/serving -mcr.microsoft.com/onnxruntime/server -gcr.io/kfserving/sklearnserver -gcr.io/kfserving/xgbserver -gcr.io/kfserving/pytorchserver -nvcr.io/nvidia/tritonserver +gcr.io/kfserving/logger:v0.4.1 +tensorflow/serving:1.14.0 +tensorflow/serving:1.14.0-gpu +mcr.microsoft.com/onnxruntime/server:v1.0.0 +gcr.io/kfserving/sklearnserver:v0.4.0 +gcr.io/kfserving/xgbserver:v0.4.0 +gcr.io/kfserving/pytorchserver:v0.4.0 +gcr.io/kfserving/pytorchserver:v0.4.0-gpu +nvcr.io/nvidia/tritonserver:20.03-py3 gcr.io/kfserving/storage-initializer:v0.4.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0 gcr.io/kfserving/kfserving-controller:v0.4.0 -gcr.io/knative-releases/knative.dev/serving/cmd/activator@sha256:8e606671215cc029683e8cd633ec5de9eabeaa6e9a4392ff289883304be1f418 -gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler-hpa@sha256:5e0fadf574e66fb1c893806b5c5e5f19139cc476ebf1dff9860789fe4ac5f545 -gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler@sha256:ef1f01b5fb3886d4c488a219687aac72d28e72f808691132f658259e4e02bb27 -gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:5ca13e5b3ce5e2819c4567b75c0984650a57272ece44bc1dabf930f9fe1e19a1 -gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio@sha256:727a623ccb17676fae8058cb1691207a9658a8d71bc7603d701e23b1a6037e6c -gcr.io/knative-releases/knative.dev/serving/cmd/webhook@sha256:1ef3328282f31704b5802c1136bd117e8598fd9f437df8209ca87366c5ce9fcb -gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb +gcr.io/knative-releases/knative.dev/serving/cmd/activator:v0.11.1 +gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler-hpa:v0.11.1 +gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler:v0.11.1 +gcr.io/knative-releases/knative.dev/serving/cmd/controller:v0.11.1 +gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio:v0.11.1 +gcr.io/knative-releases/knative.dev/serving/cmd/webhook:v0.11.1 +gcr.io/knative-releases/knative.dev/serving/cmd/queue:v0.11.1 metacontroller/metacontroller:v0.3.0 gcr.io/kubeflow-images-public/metadata:v0.1.11 gcr.io/ml-pipeline/envoy:metadata-grpc @@ -71,3 +73,8 @@ gcr.io/google_containers/spartakus-amd64:v1.1.0 tensorflow/tensorflow:1.8.0 gcr.io/kubeflow-images-public/tf_operator:v1.0.0-g92389064 gcr.io/kubeflow-images-public/admission-webhook:v1.0.0-gaf96e4e3 +gcr.io/kaniko-project/executor:v0.22.0 +brightfly/kubeflow-jupyter-lab:tf2.0-cpu +brightfly/kubeflow-jupyter-lab:tf2.0-gpu +tmaxcloudck/kubeflow-jupyter-lab:v0.1 +brightfly/fminst-webui:latest diff --git a/Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2.yaml b/Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2_local.yaml similarity index 96% rename from Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2.yaml rename to Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2_local.yaml index c1eb3d1d..42f9a026 100644 --- a/Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2.yaml +++ b/Kubeflow/kfctl_hypercloud_kubeflow.v1.0.2_local.yaml @@ -1,7 +1,9 @@ apiVersion: kfdef.apps.kubeflow.org/v1 kind: KfDef metadata: + clusterName: kubernetes creationTimestamp: null + name: kubeflow namespace: kubeflow spec: applications: @@ -147,6 +149,7 @@ spec: overlays: - istio - application + - hypercloud repoRef: name: manifests path: jupyter/notebook-controller @@ -335,8 +338,13 @@ spec: name: manifests path: seldon/seldon-core-operator name: seldon-core-operator - repos: - - name: manifests - uri: https://github.com/tmax-cloud/kubeflow-manifests/archive/kubeflow-manifests-v1.0.2.tar.gz + - kustomizeConfig: + parameters: + - name: namespace + value: monitoring + repoRef: + name: manifests + path: monitoring-resource + name: monitoring-resource version: v1.0.2 status: {} diff --git a/Kubeflow/kustomize-apply.sh b/Kubeflow/kustomize-apply.sh deleted file mode 100644 index b77c61e1..00000000 --- a/Kubeflow/kustomize-apply.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash - -dir="" - -if [ $# -eq 1 ]; then - dir=$1 -else - echo "[$0] ERROR!! Invalid argument count" - echo "[$0] [Usage] $0 ${KF_DIR}/kustomize" - exit 1 -fi - -module_num=$(ls ${dir} | wc -l) -echo "[$0] The number of modules: ${module_num}" - -i=1 -ls ${dir} | while read line -do - echo "[$0] [ ${i} / ${module_num} ] $line" - let i+=1 - - kubectl apply -k ${dir}/$line -done - -echo "[$0] Done" diff --git a/Kubeflow/kustomize.tar.gz b/Kubeflow/kustomize.tar.gz deleted file mode 100644 index 5abb2de5..00000000 Binary files a/Kubeflow/kustomize.tar.gz and /dev/null differ diff --git a/Kubeflow/kustomize_local.tar.gz b/Kubeflow/kustomize_local.tar.gz new file mode 100644 index 00000000..c72a6d6a Binary files /dev/null and b/Kubeflow/kustomize_local.tar.gz differ diff --git a/Kubeflow/sed.sh b/Kubeflow/sed.sh index 32d90cc8..fc2382b5 100644 --- a/Kubeflow/sed.sh +++ b/Kubeflow/sed.sh @@ -14,18 +14,18 @@ fi echo "[$0] Modify images in Kustomize manifest files" -sed -i "s/image: gcr.io\/ml-pipeline\/api-server/image: ${registry}\/gcr.io\/ml-pipeline\/api-server/g" ${dir}/api-service/base/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/kubernetes-sigs\/application/image: ${registry}\/cr.io\/kubeflow-images-public\/kubernetes-sigs\/application/g" ${dir}/application/base/stateful-set.yaml +sed -i "s/newName: gcr.io\/ml-pipeline\/api-server/newName: ${registry}\/gcr.io\/ml-pipeline\/api-server/g" ${dir}/api-service/base/kustomization.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/kubernetes-sigs\/application/newName: ${registry}\/gcr.io\/kubeflow-images-public\/kubernetes-sigs\/application/g" ${dir}/application/base/kustomization.yaml sed -i "s/image: argoproj\/argocli/image: ${registry}\/argoproj\/argocli/g" ${dir}/argo/base/deployment.yaml sed -i "s/image: argoproj\/workflow-controller/image: ${registry}\/argoproj\/workflow-controller/g" ${dir}/argo/base/deployment.yaml sed -i "s/image: gcr.io\/kubeflow-images-public\/ingress-setup/image: ${registry}\/gcr.io\/kubeflow-images-public\/ingress-setup/g" ${dir}/bootstrap/base/stateful-set.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/centraldashboard/image: ${registry}\/gcr.io\/kubeflow-images-public\/centraldashboard/g" ${dir}/centraldashboard/base/deployment.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/centraldashboard/newName: ${registry}\/gcr.io\/kubeflow-images-public\/centraldashboard/g" ${dir}/centraldashboard/base/kustomization.yaml sed -i "s/image: \"quay.io\/jetstack\/cert-manager-cainjector/image: \"${registry}\/quay.io\/jetstack\/cert-manager-cainjector/g" ${dir}/cert-manager/base/deployment.yaml sed -i "s/image: \"quay.io\/jetstack\/cert-manager-webhook/image: \"${registry}\/quay.io\/jetstack\/cert-manager-webhook/g" ${dir}/cert-manager/base/deployment.yaml sed -i "s/image: \"quay.io\/jetstack\/cert-manager-controller/image: \"${registry}\/quay.io\/jetstack\/cert-manager-controller/g" ${dir}/cert-manager/base/deployment.yaml sed -i "s/image: \"docker.io\/istio\/proxyv2/image: \"${registry}\/docker.io\/istio\/proxyv2/g" ${dir}/kfserving-gateway\/base\/deployment.yaml sed -i "s/image: \"docker.io\/istio\/proxyv2/image: \"${registry}\/docker.io\/istio\/proxyv2/g" ${dir}/cluster-local-gateway\/base\/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/jupyter-web-app/image: ${registry}\/gcr.io\/kubeflow-images-public\/jupyter-web-app/g" ${dir}/jupyter-web-app\/base\/deployment.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/jupyter-web-app/newName: ${registry}\/gcr.io\/kubeflow-images-public\/jupyter-web-app/g" ${dir}/jupyter-web-app\/base\/kustomization.yaml sed -i "s/gcr.io\/kubeflow-images-public\/tensorflow-1.14.0-notebook-cpu/${registry}\/gcr.io\/kubeflow-images-public\/tensorflow-1.14.0-notebook-cpu/g" ${dir}/jupyter-web-app/base/config-map.yaml sed -i "s/gcr.io\/kubeflow-images-public\/tensorflow-1.15.2-notebook-cpu/${registry}\/gcr.io\/kubeflow-images-public\/tensorflow-1.15.2-notebook-cpu/g" ${dir}/jupyter-web-app/base/config-map.yaml sed -i "s/gcr.io\/kubeflow-images-public\/tensorflow-1.15.2-notebook-gpu/${registry}\/gcr.io\/kubeflow-images-public\/tensorflow-1.15.2-notebook-gpu/g" ${dir}/jupyter-web-app/base/config-map.yaml @@ -41,12 +41,12 @@ sed -i "s/\"image\": \"gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/suggesti sed -i "s/\"image\": \"gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/suggestion-hyperopt/\"image\": \"${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/suggestion-hyperopt/g" ${dir}/katib-controller\/base\/katib-configmap.yaml sed -i "s/\"image\": \"gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/suggestion-nasrl/\"image\": \"${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/suggestion-nasrl/g" ${dir}/katib-controller\/base\/katib-configmap.yaml sed -i "s/image: docker.io\/kubeflowkatib\/mxnet-mnist/image: ${registry}\/docker.io\/kubeflowkatib\/mxnet-mnist/g" ${dir}/katib-controller\/base\/trial-template-configmap.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-controller/image: ${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-controller/g" ${dir}/katib-controller\/base\/katib-controller-deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-db-manager/image: ${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-db-manager/g" ${dir}/katib-controller\/base\/katib-db-manager-deployment.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-controller/newName: ${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-controller/g" ${dir}/katib-controller/base/kustomization.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-db-manager/newName: ${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-db-manager/g" ${dir}/katib-controller/base/kustomization.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-ui/newName: ${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-ui/g" ${dir}/katib-controller/base/kustomization.yaml sed -i "s/image: mysql/image: ${registry}\/mysql/g" ${dir}/katib-controller\/base\/katib-mysql-deployment.yaml sed -i "s/image: mysql/image: ${registry}\/mysql/g" ${dir}/mysql\/base\/deployment.yaml sed -i "s/image: mysql/image: ${registry}\/mysql/g" ${dir}/metadata\/overlays\/db\/metadata-db-deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-ui/image: ${registry}\/gcr.io\/kubeflow-images-public\/katib\/v1alpha3\/katib-ui/g" ${dir}/katib-controller\/base\/katib-ui-deployment.yaml sed -i "s/image: \"docker.io\/istio\/proxyv2/image: \"${registry}\/docker.io\/istio\/proxyv2/g" ${dir}/kfserving-gateway\/base\/deployment.yaml sed -i "s/image: \"docker.io\/istio\/proxyv2/image: \"${registry}\/docker.io\/istio\/proxyv2/g" ${dir}/cluster-local-gateway\/base\/deployment.yaml sed -i "s/\"image\" : \"gcr.io\/kfserving\/batcher/\"image\" : \"${registry}\/gcr.io\/kfserving\/batcher/g" ${dir}/kfserving-install\/base\/config-map.yaml @@ -57,11 +57,12 @@ sed -i "s/\"image\": \"mcr.microsoft.com\/onnxruntime\/server/\"image\": \"${reg sed -i "s/\"image\": \"gcr.io\/kfserving\/sklearnserver/\"image\": \"${registry}\/gcr.io\/kfserving\/sklearnserver/g" ${dir}/kfserving-install\/base\/config-map.yaml sed -i "s/\"image\": \"gcr.io\/kfserving\/xgbserver/\"image\": \"${registry}\/gcr.io\/kfserving\/xgbserver/g" ${dir}/kfserving-install\/base\/config-map.yaml sed -i "s/\"image\": \"gcr.io\/kfserving\/pytorchserver/\"image\": \"${registry}\/gcr.io\/kfserving\/pytorchserver/g" ${dir}/kfserving-install\/base\/config-map.yaml -sed -i "s/\"image\": \"nvcr.io\/nvidia\/tensorrtserver/\"image\": \"${registry}\/nvcr.io\/nvidia\/tensorrtserver/g" ${dir}/kfserving-install\/base\/config-map.yaml +sed -i "s/\"image\": \"nvcr.io\/nvidia\/tritonserver/\"image\": \"${registry}\/nvcr.io\/nvidia\/tritonserver/g" ${dir}/kfserving-install\/base\/config-map.yaml sed -i "s/\"image\" : \"gcr.io\/kfserving\/storage-initializer/\"image\" : \"${registry}\/gcr.io\/kfserving\/storage-initializer/g" ${dir}/kfserving-install\/base\/config-map.yaml +sed -i "s/image: gcr.io\/kfserving\/kfserving-controller/image: ${registry}\/gcr.io\/kfserving\/kfserving-controller/g" ${dir}/kfserving-install\/base\/statefulset.yaml sed -i "s/image: gcr.io\/kubebuilder\/kube-rbac-proxy/image: ${registry}\/gcr.io\/kubebuilder\/kube-rbac-proxy/g" ${dir}/kfserving-install\/base\/statefulset.yaml sed -i "s/image: tmaxcloudck\/hypercloud-kfserving/image: ${registry}\/tmaxcloudck\/hypercloud-kfserving/g" ${dir}/kfserving-install\/base\/statefulset.yaml -sed -i "s/image: gcr.io\/knative-releases\/knative.dev\/serving\/cmd/image: ${registry}\/gcr.io\/knative-releases\/knative.dev\/serving\/cmd/g" ${dir}/knative-install/base/deployment.yaml +sed -i "s/newName: gcr.io\/knative-releases\/knative.dev\/serving\/cmd/newName: ${registry}\/gcr.io\/knative-releases\/knative.dev\/serving\/cmd/g" ${dir}/knative-install/base/kustomization.yaml sed -i "s/image: gcr.io\/knative-releases\/knative.dev\/serving\/cmd/image: ${registry}\/gcr.io\/knative-releases\/knative.dev\/serving\/cmd/g" ${dir}/knative-install/base/image.yaml sed -i "s/image: metacontroller\/metacontroller/image: ${registry}\/metacontroller\/metacontroller/g" ${dir}/metacontroller\/base\/stateful-set.yaml sed -i "s/image: gcr.io\/kubeflow-images-public\/metadata-frontend/image: ${registry}\/gcr.io\/kubeflow-images-public\/metadata-frontend/g" ${dir}/metadata\/base\/metadata-ui-deployment.yaml @@ -71,20 +72,19 @@ sed -i "s/image: gcr.io\/tfx-oss-public\/ml_metadata_store_server/image: ${regis sed -i "s/image: gcr.io\/kubeflow-images-public\/metadata-frontend/image: ${registry}\/gcr.io\/kubeflow-images-public\/metadata-frontend/g" ${dir}/metadata\/base\/metadata-ui-deployment.yaml sed -i "s/image: minio\/minio/image: ${registry}\/minio\/minio/g" ${dir}/minio\/base\/deployment.yaml sed -i "s/image: tmaxcloudck\/notebook-controller-go:b0.0.2/image: ${registry}\/tmaxcloudck\/notebook-controller-go:b0.0.2/g" ${dir}/notebook-controller\/base\/deployment.yaml -sed -i "s/image: gcr.io\/ml-pipeline\/persistenceagent/image: ${registry}\/gcr.io\/ml-pipeline\/persistenceagent/g" ${dir}/persistent-agent\/base\/deployment.yaml -sed -i "s/image: gcr.io\/ml-pipeline\/frontend/image: ${registry}\/gcr.io\/ml-pipeline\/frontend/g" ${dir}/pipelines-ui\/base\/deployment.yaml -sed -i "s/image: gcr.io\/ml-pipeline\/viewer-crd-controller/image: ${registry}\/gcr.io\/ml-pipeline\/viewer-crd-controller/g" ${dir}/pipelines-viewer\/base\/deployment.yaml -sed -i "s/image: gcr.io\/ml-pipeline\/visualization-server/image: ${registry}\/gcr.io\/ml-pipeline\/visualization-server/g" ${dir}/pipeline-visualization-service\/base\/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/profile-controller/image: ${registry}\/gcr.io\/kubeflow-images-public\/profile-controller/g" ${dir}/profiles\/base\/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/kfam/image: ${registry}\/gcr.io\/kubeflow-images-public\/kfam/g" ${dir}/profiles\/base\/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/pytorch-operator/image: ${registry}\/gcr.io\/kubeflow-images-public\/pytorch-operator/g" ${dir}/pytorch-operator\/base\/deployment.yaml -sed -i "s/image: gcr.io\/ml-pipeline\/scheduledworkflow/image: ${registry}\/gcr.io\/ml-pipeline\/scheduledworkflow/g" ${dir}/scheduledworkflow\/base\/deployment.yaml +sed -i "s/newName: gcr.io\/ml-pipeline\/persistenceagent/newName: ${registry}\/gcr.io\/ml-pipeline\/persistenceagent/g" ${dir}/persistent-agent/base/kustomization.yaml +sed -i "s/newName: gcr.io\/ml-pipeline\/frontend/newName: ${registry}\/gcr.io\/ml-pipeline\/frontend/g" ${dir}/pipelines-ui/base/kustomization.yaml +sed -i "s/newName: gcr.io\/ml-pipeline\/viewer-crd-controller/newName: ${registry}\/gcr.io\/ml-pipeline\/viewer-crd-controller/g" ${dir}/pipelines-viewer/base/kustomization.yaml +sed -i "s/newName: gcr.io\/ml-pipeline\/visualization-server/newName: ${registry}\/gcr.io\/ml-pipeline\/visualization-server/g" ${dir}/pipeline-visualization-service/base/kustomization.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public/newName: ${registry}\/gcr.io\/kubeflow-images-public/g" ${dir}/profiles/base/kustomization.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/pytorch-operator/newName: ${registry}\/gcr.io\/kubeflow-images-public\/pytorch-operator/g" ${dir}/pytorch-operator/base/kustomization.yaml +sed -i "s/newName: gcr.io\/ml-pipeline\/scheduledworkflow/newName: ${registry}\/gcr.io\/ml-pipeline\/scheduledworkflow/g" ${dir}/scheduledworkflow/base/kustomization.yaml sed -i "s/image: 'docker.io\/seldonio\/seldon-core-operator/image: '${registry}\/docker.io\/seldonio\/seldon-core-operator/g" ${dir}/seldon-core-operator\/base\/resources.yaml sed -i "s/image: gcr.io\/spark-operator\/spark-operator/image: ${registry}\/gcr.io\/spark-operator\/spark-operator/g" ${dir}/spark-operator\/base\/deploy.yaml sed -i "s/image: gcr.io\/spark-operator\/spark-operator/image: ${registry}\/gcr.io\/spark-operator\/spark-operator/g" ${dir}/spark-operator\/base\/crd-cleanup-job.yaml sed -i "s/image: gcr.io\/google_containers\/spartakus-amd64/image: ${registry}\/gcr.io\/google_containers\/spartakus-amd64/g" ${dir}/spartakus\/base\/deployment.yaml sed -i "s/image: tensorflow\/tensorflow/image: ${registry}\/tensorflow\/tensorflow/g" ${dir}/tensorboard\/base\/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/tf_operator/image: ${registry}\/gcr.io\/kubeflow-images-public\/tf_operator/g" ${dir}/tf-job-operator\/base\/deployment.yaml -sed -i "s/image: gcr.io\/kubeflow-images-public\/admission-webhook/image: ${registry}\/gcr.io\/kubeflow-images-public\/admission-webhook/g" ${dir}/webhook\/base\/deployment.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/tf_operator/newName: ${registry}\/gcr.io\/kubeflow-images-public\/tf_operator/g" ${dir}/tf-job-operator/base/kustomization.yaml +sed -i "s/newName: gcr.io\/kubeflow-images-public\/admission-webhook/newName: ${registry}\/gcr.io\/kubeflow-images-public\/admission-webhook/g" ${dir}/webhook/base/kustomization.yaml echo "[$0] Done" diff --git a/Package/README.md b/Package/README.md index ed22f517..fe1b002f 100644 --- a/Package/README.md +++ b/Package/README.md @@ -2,7 +2,7 @@ # OS 설치 & package repo 구축 가이드 ## 구성 요소 및 버전 -* HyperCloud 패키지(ck-ftp@192.168.1.150:/home/ck-ftp/k8s_pl/install/offline/archive_20.07.10) +* HyperCloud 패키지(ck-ftp@192.168.1.150:/home/ck-ftp/k8s_pl/install/offline/archive_20.08.03) * ISO 파일(CentOS 7.7 :http://vault.centos.org/7.7.1908/isos/x86_64/ 또는 http://192.168.2.136/ISOs/CentOS-7-x86_64-DVD-1908.iso) ## Prerequisites @@ -46,8 +46,8 @@ * 목적 : `폐쇄망일 때 yum repository 구축` * 생성 순서 : * 패키지 가져오기 - * scp -r ck-ftp@192.168.1.150:/home/ck-ftp/k8s_pl/install/offline/archive_20.07.10 . - * cp -rT ./archive_20.07.10 /tmp/localrepo + * scp -r ck-ftp@192.168.1.150:/home/ck-ftp/k8s_pl/install/offline/archive_20.08.03 . + * cp -rT ./archive_20.08.03 /tmp/localrepo * CentOS Repository 비활성화 * sudo vi /etc/yum.repos.d/CentOS-Base.repo * [base], [updates], [extra] repo config 에 enabled=0 추가 diff --git a/Pod_GPU plugin/nvidia-device-plugin/README.md b/Pod_GPU plugin/nvidia-device-plugin/README.md index b5c79161..6f460dee 100644 --- a/Pod_GPU plugin/nvidia-device-plugin/README.md +++ b/Pod_GPU plugin/nvidia-device-plugin/README.md @@ -47,6 +47,7 @@ ## Step 0. NVIDIA driver 설치 * 목적 : `GPU device에 적절한 nvidia driver를 설치` +* 버전 요구사항: `361.93 이상` * 생성 순서 : * nouveau 비활성화 ```bash @@ -73,6 +74,29 @@ ```bash $ nvidia-smi ``` +### 참고용) Nvidia Driver Upgrade 방법 +- Step 1: nvidia-device-plugin 삭제 + ```bash + $ cd ${INSTALLER_HOME} + $ kubectl delete -f ./nvidia-device-plugin-daemonset.yml + ``` +- Step 2: Nvidia GPU Driver 삭제 후 재부팅 + ```bash + $ # Nvidia GPU Driver install 파일 준비 (run파일) + $ chmod +x ./{your-nvidia-file.run} + $ ./{your-nvidia-file.run} --uninstall + $ reboot + ``` +- Step 3: Nvidia GPU Driver 재설치 후 확인 + ```bash + $ chmod +x ./{your-nvidia-file.run} + $ ./{your-nvidia-file.run} --dkms -s + ``` +- Step 4: Nvidia Device Plugin 재 설치 + ```bash + $ cd ${INSTALLER_HOME} + $ kubectl create -f ./nvidia-device-plugin-daemonset.yml + ``` ## Step 1. 필요한 패키지 설치 및 설정 ### Docker를 사용하는 경우 diff --git a/README.md b/README.md index 0776a8ff..5bd6bc82 100644 --- a/README.md +++ b/README.md @@ -4,34 +4,33 @@ ### Module (Required) | Module | Version | Guide | 진행률(O/△/X) | | ------ | ------ | ------ | ------ | -| CentOS 설치 & package repo | 7.7 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Package/README.md | O | -| Image registry | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Image_Registry/README.md | O | -| K8s Master | v1.17.6 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/K8S_Master/README.md | O | -| K8s Worker | v1.17.6 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/K8S_Worker/README.md | O | -| CNI | | https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/CNI | O | -| MetalLB | | https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/MetalLB | O | +| CentOS 설치 & package repo | 7.7 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/Package/README.md | O | +| Image registry | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/Image_Registry/README.md | O | +| K8s Master | v1.17.6 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/K8S_Master/README.md | O | +| K8s Worker | v1.17.6 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/K8S_Worker/README.md | O | +| CNI | | https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/CNI | O | +| MetalLB | v0.8.2 | https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/MetalLB | O | | Rook Ceph | v1.3.6 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/rook-ceph/README.md | O | -| HyperAuth | 1.0.5.6 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperAuth/README.md | O | -| HyperCloud Operator | v4.1.1.0+ | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Operator/v4.1.1.0/README.md | O | -| HyperCloud Webhook | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/HyperCloud%20Webhook/README.md | O | -| Prometheus | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Prometheus/README.md | O | -| Console | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Console/README.md | O | -| Tekton | v0.12.1+ | https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/Tekton_CI_CD | O | -| Catalog Controller | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/CatalogController/README.md | O | -| TemplateServiceBroker | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/TemplateServiceBroker/README.md | O | -| SecretWatcher | v4.1.0.9 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/SecretWatcher/README.md | O | +| HyperCloud Operator | v4.1.0.13 ~ v4.1.0.40 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Operator/v4.1.0.13/README.md | O | +| HyperCloud Webhook | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/HyperCloud%20Webhook/README.md | O | +| Prometheus | v2.11.0 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/Prometheus/README.md | O | +| Console | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/Console/README.md | O | +| Tekton | v0.12.1+ | https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/Tekton_CI_CD | O | +| Catalog Controller | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/CatalogController/README.md | O | +| TemplateServiceBroker | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/TemplateServiceBroker/README.md | O | +| SecretWatcher | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/SecretWatcher/README.md | O | ### Module (Optional) | Module | Version | Guide | 진행률(O/△/X) | | ------ | ------ | ------ | ------ | -| NetworkAgent | | https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/NetworkAgent | O | -| Pod_GPU plugin | |
  • https://github.com/tmax-cloud/hypercloud-install-guide/tree/4.1/Pod_GPU%20plugin
  • NVIDIA Device Plugin : https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Pod_GPU%20plugin/nvidia-device-plugin/README.md
  • NVIDIA Pod GPU Metrics Exporter : https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Pod_GPU%20plugin/nvidia-pod-gpu-metrics-exporter/README.md
| O | -| Istio | | [installation guide](https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Istio/README.md) | O | -| Kubeflow | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Kubeflow/README.md | O | -| EFK | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/EFK/README.md | O | +| NetworkAgent | | https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/NetworkAgent | O | +| Pod_GPU plugin | |
  • https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/Pod_GPU%20plugin
  • NVIDIA Device Plugin : https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/Pod_GPU%20plugin/nvidia-device-plugin/README.md
  • NVIDIA Pod GPU Metrics Exporter : https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/Pod_GPU%20plugin/nvidia-pod-gpu-metrics-exporter/README.md
| O | +| Istio | 1.5.1 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Istio/README.md | O | +| Kubeflow | | https://github.com/tmax-cloud/install-ai-devops/tree/4.1 | O | +| EFK | E(7.2.0), F(v1.4.2), K(7.2.0) | https://github.com/tmax-cloud/hypercloud-install-guide/blob/master/EFK/README.md | O | | Multicloud-console | | | X | | Capi provider | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Capi/README.md | O | -| NGINX Ingress Controller | 0.33.0 | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/IngressNginx/README.md | O | +| NGINX Ingress Controller | 0.33.0 | https://github.com/tmax-cloud/hypercloud-install-guide/tree/master/IngressNginx/system | O | | kubefed | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Fed/README.md | O | | Grafana | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Grafana/README.md | O | | Helm | | https://github.com/tmax-cloud/hypercloud-install-guide/blob/4.1/Helm/README.md | O | diff --git a/SecretWatcher/README.md b/SecretWatcher/README.md index efab0587..e953416f 100644 --- a/SecretWatcher/README.md +++ b/SecretWatcher/README.md @@ -81,7 +81,7 @@ ``` * image version 수정 ```bash - $ sed -i 's/tmaxcloudck\/hypercloud4-secret-watcher:latest/tmaxcloudck\/hypercloud4-secret-watcher:'${HPCD_SW_VERSION}'/g' ${HPCD_SW_HOME}/secret-watcher-${HPCD_SW_VERSION}/k8s-install/hypercloud-secret-watcher-daemonset.yaml + $ sed -i 's/tmaxcloudck\/hypercloud4-secret-watcher:latest/tmaxcloudck\/hypercloud4-secret-watcher:'b${HPCD_SW_VERSION}'/g' ${HPCD_SW_HOME}/secret-watcher-${HPCD_SW_VERSION}/k8s-install/hypercloud-secret-watcher-daemonset.yaml ``` * 비고 * 폐쇄망의 경우 diff --git a/Tekton_CI_CD/README.md b/Tekton_CI_CD/README.md index 10a000cb..d29339e4 100644 --- a/Tekton_CI_CD/README.md +++ b/Tekton_CI_CD/README.md @@ -6,8 +6,8 @@ | ------ | --- | ------ | | Pipeline | v0.12.1 | [Tekton Pipeline Installation Guide](./pipeline.md) | | Trigger | v0.4.0 | [Tekton Trigger Installation Guide](./trigger.md) | -| Approval | 0.0.3 | [Approval for CI/CD Installation Guide](https://github.com/tmax-cloud/approval-watcher/blob/doc-temp/docs/installation.md) | -| Mail-notifier | v0.0.4 | [Mail notifier Installation Guide](https://github.com/cqbqdd11519/mail-notifier/blob/master/docs/installation.md) | +| Approval | 0.0.3 | [Approval for CI/CD Installation Guide](./approval.md) | +| Mail-notifier | v0.0.4 | [Mail notifier Installation Guide](./mail-notifier.md) | | CI/CD Templates | 1.1.4 | [CI/CD Templates Installation Guide](https://github.com/tmax-cloud/hypercloud-operator/blob/master/_catalog_museum/was/installation.md) | ### Usage Guide diff --git a/Tekton_CI_CD/approval.md b/Tekton_CI_CD/approval.md new file mode 100644 index 00000000..7807c042 --- /dev/null +++ b/Tekton_CI_CD/approval.md @@ -0,0 +1,100 @@ +# Approval for CI/CD 설치 가이드 + +## 구성 요소 및 버전 +* approval-watcher ([tmaxcloudck/approval-watcher:0.0.3](https://hub.docker.com/layers/tmaxcloudck/approval-watcher/0.0.3/images/sha256-6f5fd3fbe9f45909954181a9121321bbf13dc5f46724a6ad72bb156754cac2c6?context=explore)) +* approval-step-server ([tmaxcloudck/approval-step-server:0.0.3](https://hub.docker.com/layers/tmaxcloudck/approval-step-server/0.0.3/images/sha256-dd1eca762c7009676c2ae41d409ee084e803eefe40581ad95463f20a88bc0a59?context=explore)) + +## Prerequisites + +## 폐쇄망 설치 가이드 +설치를 진행하기 전 아래의 과정을 통해 필요한 이미지 및 yaml 파일을 준비한다. +1. 폐쇄망에서 설치하는 경우 사용하는 image repository에 Approval-watcher 설치 시 필요한 이미지를 push한다. + * 작업 디렉토리 생성 및 환경 설정 + ```bash + mkdir -p $HOME/approval-install + cd $HOME/approval-install + ``` + * 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다. + ```bash + # Approval 필수 이미지 Pull + docker pull tmaxcloudck/approval-watcher:0.0.3 + docker pull tmaxcloudck/approval-step-server:0.0.3 + + # 이미지 태그 + docker tag tmaxcloudck/approval-watcher:0.0.3 approval-watcher:0.0.3 + docker tag tmaxcloudck/approval-step-server:0.0.3 approval-step-server:0.0.3 + + # Approval 필수 이미지 Save + docker save approval-watcher:0.0.3 > approval-watcher-0.0.3.tar + docker save approval-step-server:0.0.3 > approval-step-server-0.0.3.tar + ``` + * install yaml을 다운로드한다. + ```bash + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/crds/tmax.io_approvals_crd.yaml crd.yaml + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/namespace.yaml + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/service_account.yaml + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/role.yaml + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/role_binding.yaml + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/service.yaml + wget https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/proxy-server.yaml + ``` + +2. 폐쇄망 환경으로 전송 + ```bash + # 생성된 파일 모두 SCP 또는 물리 매체를 통해 폐쇄망 환경으로 복사 + scp -r $HOME/approval-install : + ``` + +3. 위의 과정에서 생성한 tar 파일들을 폐쇄망 환경으로 이동시킨 뒤 사용하려는 registry에 이미지를 push한다. + ```bash + # 이미지 레지스트리 주소 + REGISTRY=[IP:PORT] + + cd + + # Load images + docker load < approval-watcher-0.0.3.tar + docker load < approval-step-server-0.0.3.tar + + # Tag images + docker tag approval-watcher:0.0.3 $REGISTRY/approval-watcher:0.0.3 + docker tag approval-step-server:0.0.3 $REGISTRY/approval-step-server:0.0.3 + + # Push images + docker push $REGISTRY/approval-watcher:0.0.3 + docker push $REGISTRY/approval-step-server:0.0.3 + ``` +4. YAML 수정 + ```bash + REGISTRY=[IP:PORT] + + cp proxy-server.yaml updated.yaml + sed -i "s/tmaxcloudck\/approval-watcher:latest/$REGISTRY\/approval-watcher:0.0.3/g" updated.yaml + ``` + +## Install Steps +1. [Approval 설치](#step-1-approval-설치) + +## Step 1. Approval 설치 +* 목적 : `단계 별 승인에 필요한 구성 요소 설치` +* 생성 순서 : 아래 command로 설치 yaml 적용 + * (외부망 연결된 환경 설치 시 실행) + ```bash + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/crds/tmax.io_approvals_crd.yaml + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/namespace.yaml + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/service_account.yaml + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/role.yaml + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/role_binding.yaml + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/service.yaml + kubectl apply -f https://raw.githubusercontent.com/tmax-cloud/approval-watcher/master/deploy/proxy-server.yaml + ``` + * (폐쇄망 환경 설치 시 실행) + ```bash + kubectl apply -f crd.yaml + kubectl apply -f namespace.yaml + kubectl apply -f service_account.yaml + kubectl apply -f role.yaml + kubectl apply -f role_binding.yaml + kubectl apply -f service.yaml + kubectl apply -f updated.yaml + ``` diff --git a/Tekton_CI_CD/mail-notifier.md b/Tekton_CI_CD/mail-notifier.md new file mode 100644 index 00000000..6477e3d0 --- /dev/null +++ b/Tekton_CI_CD/mail-notifier.md @@ -0,0 +1,119 @@ +# Mail-Notifier for CI/CD 설치 가이드 + +## 구성 요소 및 버전 +* mail-sender-server ([tmaxcloudck/mail-sender-server:v0.0.4](https://hub.docker.com/layers/tmaxcloudck/mail-sender-server/v0.0.4/images/sha256-3d87f419d056132690bd7cdcb5aab1abe0021ae12b4efd50a8b7c0be7a44dd86?context=explore)) +* mail-sender-client ([tmaxcloudck/mail-sender-client:v0.0.4](https://hub.docker.com/layers/tmaxcloudck/mail-sender-client/v0.0.4/images/sha256-0364005e432a67e839cee04cdb0ebb5d925eb4427fd248f346566300f890d046?context=explore)) + +## Prerequisites + +## 폐쇄망 설치 가이드 +설치를 진행하기 전 아래의 과정을 통해 필요한 이미지 및 yaml 파일을 준비한다. +1. 폐쇄망에서 설치하는 경우 사용하는 image repository에 Approval-watcher 설치 시 필요한 이미지를 push한다. + * 작업 디렉토리 생성 및 환경 설정 + ```bash + mkdir -p $HOME/mail-install + cd $HOME/mail-install + ``` + * 외부 네트워크 통신이 가능한 환경에서 필요한 이미지를 다운받는다. + ```bash + # Mail Notifier 필수 이미지 Pull + docker pull tmaxcloudck/mail-sender-server:v0.0.4 + docker pull tmaxcloudck/mail-sender-client:v0.0.4 + + # 이미지 태그 + docker tag tmaxcloudck/mail-sender-server:v0.0.4 mail-sender-server:v0.0.4 + docker tag tmaxcloudck/mail-sender-client:v0.0.4 mail-sender-client:v0.0.4 + + # Mail Notifier 필수 이미지 Save + docker save mail-sender-server:v0.0.4 > mail-sender-server-v0.0.4.tar + docker save mail-sender-client:v0.0.4 > mail-sender-client-v0.0.4.tar + ``` + * install yaml을 다운로드한다. + ```bash + wget https://raw.githubusercontent.com/cqbqdd11519/mail-notifier/master/deploy/service.yaml + wget https://raw.githubusercontent.com/cqbqdd11519/mail-notifier/master/deploy/server.yaml + wget https://raw.githubusercontent.com/cqbqdd11519/mail-notifier/master/deploy/secret.yaml.template + ``` + +2. 폐쇄망 환경으로 전송 + ```bash + # 생성된 파일 모두 SCP 또는 물리 매체를 통해 폐쇄망 환경으로 복사 + scp -r $HOME/mail-install : + ``` + +3. 위의 과정에서 생성한 tar 파일들을 폐쇄망 환경으로 이동시킨 뒤 사용하려는 registry에 이미지를 push한다. + ```bash + # 이미지 레지스트리 주소 + REGISTRY=[IP:PORT] + + cd + + # Load images + docker load < mail-sender-server-v0.0.4.tar + docker load < mail-sender-client-v0.0.4.tar + + # Tag images + docker tag mail-sender-server:v0.0.4 $REGISTRY/mail-sender-server:v0.0.4 + docker tag mail-sender-client:v0.0.4 $REGISTRY/mail-sender-client:v0.0.4 + + # Push images + docker push $REGISTRY/mail-sender-server:v0.0.4 + docker push $REGISTRY/mail-sender-client:v0.0.4 + ``` +4. YAML 수정 + ```bash + REGISTRY=[IP:PORT] + + cp server.yaml updated.yaml + sed -i "s/tmaxcloudck\/mail-sender-server:v0.0.3/$REGISTRY\/mail-sender-server:v0.0.3/g" updated.yaml + ``` + +## Install Steps +1. [SMTP 서버 설정](#step-1-smtp-서버-설정) +2. [Mail Notifier Server 설치](#step-2-Server-설치) + +## Step 1. SMTP 서버 설정 +* 목적 : `Mail-notifier에서 사용할 외부 SMTP 서버 설정` +* 생성 순서 : 아래 command로 설정 + * (외부망 연결된 환경 설치 시 실행) + ```bash + SMTP_SERVER= + SMTP_USER= + SMTP_PW= + NAMESPACE=approval-system + + curl https://raw.githubusercontent.com/cqbqdd11519/mail-notifier/master/deploy/secret.yaml.template -s | \ + sed "s//'${SMTP_SERVER}'/g" | \ + sed "s//'${SMTP_USER}'/g" | \ + sed "s//'${SMTP_PW}'/g" | \ + kubectl apply --namespace ${NAMESPACE} -f - + ``` + * (폐쇄망 환경 설치 시 실행) + ```bash + SMTP_SERVER= + SMTP_USER= + SMTP_PW= + NAMESPACE=approval-system + + cp secret.yaml.template secret.yaml + sed -i "s//'${SMTP_SERVER}'/g" secret.yaml + sed -i "s//'${SMTP_USER}'/g" secret.yaml + sed - i"s//'${SMTP_PW}'/g" secret.yaml + kubectl apply --namespace ${NAMESPACE} -f secret.yaml + ``` + +## Step 2. Server 설치 +* 목적 : `Mail-notifier 구성 요소 설치` +* 생성 순서 : 아래 command로 설치 yaml 적용 + * (외부망 연결된 환경 설치 시 실행) + ```bash + NAMESPACE=approval-system + kubectl apply --namespace ${NAMESPACE} --filename https://raw.githubusercontent.com/cqbqdd11519/mail-notifier/master/deploy/service.yaml + kubectl apply --namespace ${NAMESPACE} --filename https://raw.githubusercontent.com/cqbqdd11519/mail-notifier/master/deploy/server.yaml + ``` + * (폐쇄망 환경 설치 시 실행) + ```bash + NAMESPACE=approval-system + kubectl apply --namespace ${NAMESPACE} -f service.yaml + kubectl apply --namespace ${NAMESPACE} -f updated.yaml + ``` diff --git a/VM_KubeVirt/README.md b/VM_KubeVirt/README.md index e79bae18..9694cf9a 100644 --- a/VM_KubeVirt/README.md +++ b/VM_KubeVirt/README.md @@ -45,6 +45,9 @@ ```bash $ make upload ``` +4. docker 또는 crio의 search registries에 $REGISTRY_ENDPOINT 가 등록되어 있지 않은 경우 아래의 명령어를 통해 yaml을 수정합니다.\ + ```sed -i 's/tmaxcloudck/'$REGISTRY_ENDPOINT'\/tmaxcloudck/g' yamls/kubevirt-operator.yaml```\ + ```sed -i 's/tmaxcloudck/'$REGISTRY_ENDPOINT'\/tmaxcloudck/g' yamls/virtvnc.yaml``` ## Install Steps 1. 아래의 명령어를 수행하여 설치를 진행합니다 diff --git a/installer/install.sh b/installer/install.sh index fc96e639..ab874791 100644 --- a/installer/install.sh +++ b/installer/install.sh @@ -7,10 +7,13 @@ yaml_dir="${install_dir}/yaml" type=$2 -#sudo yum update -y - os_check=$(awk -F= '/^NAME/{print $2}' /etc/os-release) +# ProLinux == CentOS Linux +if [ ${os_check} == "\"ProLinux\"" ]; then + os_check="\"CentOS Linux\""; +fi + function set_env() { echo "=========================================================================" @@ -19,6 +22,7 @@ function set_env() { # centos if [[ ${os_check} == "\"CentOS Linux\"" ]]; then + # disable firewall sudo systemctl disable firewalld sudo systemctl stop firewalld @@ -31,16 +35,16 @@ function set_env() { sudo setenforce 0 sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config - #crio-kube set - sudo modprobe overlay + #crio-kube set + sudo modprobe overlay sudo modprobe br_netfilter - sudo cat << "EOF" | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf + sudo cat << "EOF" | sudo tee -a /etc/sysctl.d/99-kubernetes-cri.conf net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF - sudo sysctl --system + sudo sysctl --system # ubuntu elif [[ ${os_check} = "\"Ubuntu\"" ]]; then @@ -62,19 +66,8 @@ function install_crio() { echo "========================== start install crio ==========================" echo "=========================================================================" - if [[ -z ${crioVersion} ]]; then - crioVersion=1.17 - else - crioVersion=${crioVersion} - fi - #centos if [[ ${os_check} == "\"CentOS Linux\"" ]]; then - # set repo - #curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo \ - #https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo - #curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${crioVersion}.repo \ - #https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${crioVersion}/CentOS_7/devel:kubic:libcontainers:stable:cri-o:${crioVersion}.repo; # install crio sudo yum install -y cri-o @@ -103,10 +96,29 @@ function install_crio() { sudo sed -i "s|{imageRegistry}|${imageRegistry}|g" /etc/crio/crio.conf sudo sed -i "s|{imageRegistry}|${imageRegistry}|g" /etc/containers/registries.conf fi - sudo systemctl restart crio + sudo systemctl restart crio + + elif [[ ${os_check} = "\"Ubuntu\"" ]]; then + # install crio + sudo apt-get -y install cri-o-${crioVersion} + sudo systemctl enable crio.service + sudo systemctl start crio.service + + # check crio + sudo systemctl status crio + + # remove cni0 + sudo rm -rf /etc/cni/net.d/100-crio-bridge.conf + sudo rm -rf /etc/cni/net.d/200-loopback.conf + + # edit crio config + sudo systemctl restart crio + + # others else - sudo systemctl restart crio + sudo echo "This OS is not supported." + sudo exit 100 fi @@ -118,49 +130,39 @@ function install_kube() { echo "======================= start install kubernetes ======================" echo "=========================================================================" - #cat < /proc/sys/net/ipv4/ip_forward @@ -171,9 +173,9 @@ function install_kube() { sudo sed -i "s|{apiServer}|${apiServer}|g" ${yaml_dir}/kubeadm-config.yaml sudo sed -i "s|{podSubnet}|${podSubnet}|g" ${yaml_dir}/kubeadm-config.yaml if [[ "${imageRegistry}" == "" ]]; then - sudo sed -i "s|{imageRegistry}/|${imageRegistry}|g" ${yaml_dir}/kubeadm-config.yaml + sudo sed -i "s|{imageRegistry}/|${imageRegistry}|g" ${yaml_dir}/kubeadm-config.yaml else - sudo sed -i "s|{imageRegistry}|${imageRegistry}|g" ${yaml_dir}/kubeadm-config.yaml + sudo sed -i "s|{imageRegistry}|${imageRegistry}|g" ${yaml_dir}/kubeadm-config.yaml fi if [ "${type}" == "mainMaster" ];then @@ -231,4 +233,3 @@ function main(){ esac } main $1 -