Skip to content

Commit ea128e0

Browse files
lgritzlgritz
committed
Initial commit
0 parents  commit ea128e0

File tree

558 files changed

+95922
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

558 files changed

+95922
-0
lines changed

‎CREDITS

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
cue3bot
2+
3720 Matt Chambers
3+
167 John Welborn
4+
123 Michael Zhang
5+
50 J Robert Ray
6+
27 Bond-Jay Ting
7+
22 Kasra Faghihi
8+
14 Blair Zajac
9+
2 Kevin Coats
10+
1 Jordon Phillips
11+
12+
rqd
13+
473 John Welborn
14+
25 J Robert Ray
15+
6 Yudi Xue
16+
2 Blair Zajac
17+
2 Michael Zhang
18+
2 Jordon Phillips
19+
1 Kasra Faghihi
20+
21+
spi_cue
22+
252 John Welborn
23+
226 Matt Chambers
24+
12 Jordon Phillips
25+
8 Yudi Xue
26+
5 J Robert Ray
27+
4 Michael Zhang
28+
3 Blair Zajac
29+
30+
python_ice_server
31+
27 Blair Zajac
32+
8 Cottalango Leon
33+
5 J Robert Ray
34+
3 John Welborn
35+
2 Michael Zhang
36+
1 Geo Snelling
37+
1 Sam Richards

‎LICENSE

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Copyright 2018 Sony Pictures Imageworks Inc.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.

‎README

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
2+
Introduction:
3+
~~~~~~~~~~~~~
4+
5+
Cue3 is the render farm management software used at Imageworks. It allows users
6+
to submit jobs to a configurable dispatch queue that allocates the necessary
7+
computational resources. Cue3 has been in use at Imageworks since 2007.
8+
9+
Basic Queue Vocabulary:
10+
~~~~~~~~~~~~~~~~~~~~~~~
11+
12+
Queue - The renderfarm as used to actively process a large number of render
13+
jobs using define priorities.
14+
15+
Cue - A nickname for the entire queuing system. The base name of the
16+
SPI render queue tools. Thus "cuetopia3", "cuerun", "cueman3", etc.
17+
18+
Cores - The processing units that make up a render machine. A machine
19+
known as a "dub" contains 8 cpu cores.
20+
21+
Procs - Old terminology from cue2 where it use to refer to a single
22+
core of a machine. Typically a proc refers to a running frame.
23+
24+
Job - Any script sent to the queue to be processed on remote cores.
25+
26+
Layers - The sub-jobs in an outline script job.
27+
28+
Frames - An individual command contained in a layer.
29+
30+
Dependent Job - a job which will not run until the frames of
31+
another job are completed.
32+
33+
Soft Dependency - when all the frames of the first job need to
34+
finish for the second job to begin.
35+
36+
Hard Dependency - a frame to frame dependency (only the corresponding
37+
frames need to finish).
38+
39+
cuetopia3 - A graphical user interface to monitor and interact with
40+
Jobs, Layers and Frames.
41+
42+
Outline Script - A powerful tool for submitting jobs to the queue. Outline
43+
scripts can batch what would be multiple job submissions into a single job,
44+
setting up dependencies and/or running in parallel. Outline scripts can
45+
submit almost any type of job to the cue - maya, katana, even shell commands.
46+
47+
pycuerun - The command used to submit outline scripts to the queue.
48+
booked - When machine resources are assigned to a job causing a
49+
frame to start running.
50+
51+
Note:
52+
~~~~~
53+
54+
This project is missing some components that we couldn't open source and is not
55+
yet expected to build. But stay tuned for more information and development.

‎cue3bot/.gitignore

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
/build
2+
/classes-slice
3+
/classes-test
4+
/dist
5+
/logs
6+
/src-slice
7+
/target
8+
/velocity.log
9+
10+
/.idea
11+
/nbproject
12+
/NETBEANS_CLASSPATH_SPECIFIC

‎cue3bot/.gitlab-ci.yml

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
variables:
2+
GIT_SUBMODULE_STRATEGY: normal
3+
4+
test:
5+
stage: test
6+
script: bin/sbt -Dsbt.log.noformat=true clean test
7+
artifacts:
8+
paths:
9+
- target/
10+
expire_in: 1 hour
11+
12+
pack:
13+
stage: pack
14+
script: bin/sbt -Dsbt.log.noformat=true pack
15+
artifacts:
16+
paths:
17+
- target/pack
18+
expire_in: 1 week
19+
20+
stages:
21+
- test
22+
- pack

‎cue3bot/bin/check_ice.py

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/usr/bin/env python
2+
3+
4+
# Copyright (c) 2018 Sony Pictures Imageworks Inc.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
19+
import sys
20+
import socket
21+
import os
22+
23+
try:
24+
host = sys.argv[1]
25+
object = sys.argv[2]
26+
port = sys.argv[3]
27+
except:
28+
print "usage: check_ice.py host object port"
29+
sys.exit(1)
30+
31+
import Ice
32+
33+
try:
34+
ic = Ice.initialize(["--Ice.ImplicitContext=Shared", "--Ice.MessageSizeMax=10240"])
35+
implicit_context = ic.getImplicitContext()
36+
implicit_context.put('argv', 'xyzzy')
37+
implicit_context.put('hostname', str(socket.gethostname()))
38+
implicit_context.put('pid', str(os.getpid()))
39+
implicit_context.put('username', 'nagios')
40+
proxy = ic.stringToProxy("%s:default -h %s -p %s" % (object, host, port))
41+
proxy.ice_ping()
42+
except Exception, e:
43+
print e
44+
sys.exit(1)
45+
finally:
46+
ic.destroy()
47+
48+
49+
print "%s ice object is available" % object

‎cue3bot/bin/sbt

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/bin/sh
2+
3+
# ORACLE_HOME should match project/build.scala
4+
export ORACLE_HOME=/usr/lib64/oracle-instantclient-11.2
5+
export LD_LIBRARY_PATH=/shots/spi/home/java/FileSequence/x86_64/v6/rhel7/:${ORACLE_HOME}
6+
export TNS_ADMIN=/opt/tns
7+
8+
project_dir=$(readlink -f $(dirname $0)/..)
9+
10+
exec \
11+
/usr/bin/java \
12+
-Dcatalina.base=$project_dir \
13+
-XX:+UseParNewGC \
14+
-XX:+UseConcMarkSweepGC \
15+
-XX:+CMSClassUnloadingEnabled \
16+
-XX:CMSInitiatingOccupancyFraction=70 \
17+
-verbose:gc \
18+
-XX:+PrintGCDateStamps \
19+
-XX:+PrintGCDetails \
20+
-XX:+PrintGCTimeStamps \
21+
-XX:+PrintHeapAtGC \
22+
-XX:+PrintTenuringDistribution \
23+
-Xloggc:${project_dir}/logs/gc.log \
24+
-XX:+UseGCLogFileRotation \
25+
-XX:NumberOfGCLogFiles=10 \
26+
-XX:GCLogFileSize=1M \
27+
-Dfile.encoding=UTF8 \
28+
-Djava.net.preferIPv4Stack=true \
29+
-Xms3G \
30+
-Xmx3G \
31+
-XX:+CMSClassUnloadingEnabled \
32+
-XX:ReservedCodeCacheSize=192m \
33+
$EXTRA_JVM_OPTS \
34+
-jar `dirname $0`/sbt-launch.jar \
35+
"$@"

‎cue3bot/build.sbt

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
import sbtzerocice.{ZeroCIcePlugin=>Ice}
2+
import scalariform.formatter.preferences._
3+
4+
// Disable scala!
5+
crossPaths := false
6+
autoScalaLibrary := false
7+
8+
packSettings
9+
10+
patchPack
11+
12+
packMain := Map("cuebot" -> "com.imageworks.spcue.CuebotApplication")
13+
14+
packGenerateWindowsBatFile := false
15+
16+
packExcludeJars := Seq("scala-compiler-.*\\.jar")
17+
18+
packJvmOpts := Map("cuebot" -> Seq(
19+
"-server",
20+
"-Xms3g",
21+
"-Xmx3g",
22+
"-XX:MaxDirectMemorySize=9223372036854775807",
23+
"-Dnet.spy.log.LoggerImpl=net.spy.log.Log4JLogger",
24+
"-Dhttp.proxyHost=proxy.spimageworks.com",
25+
"-Dhttp.proxyPort=8080",
26+
"-Dhttps.proxyHost=proxy.spimageworks.com",
27+
"-Dhttps.proxyPort=8080",
28+
"-Djava.net.preferIPv4Stack=true",
29+
// Garbage collection logging.
30+
"-verbose:gc",
31+
"-XX:+PrintGCDateStamps",
32+
"-XX:+PrintGCDetails",
33+
"-XX:+PrintGCTimeStamps",
34+
"-XX:+PrintHeapAtGC",
35+
"-XX:+PrintTenuringDistribution",
36+
"-Xloggc:${PROG_HOME}/logs/gc.log",
37+
"-XX:+UseGCLogFileRotation",
38+
"-XX:NumberOfGCLogFiles=10",
39+
"-XX:GCLogFileSize=1M",
40+
// Garbage collection configuration.
41+
"-XX:+UseParNewGC",
42+
"-XX:+UseConcMarkSweepGC",
43+
"-XX:+CMSClassUnloadingEnabled",
44+
"-XX:CMSInitiatingOccupancyFraction=70",
45+
"-Djava.library.path=${ORACLE_HOME}:/shots/spi/home/java/FileSequence/x86_64/v6/rhel7"
46+
))
47+
48+
net.virtualvoid.sbt.graph.Plugin.graphSettings
49+
50+
genSliceSettings
51+
52+
scalariformSettings
53+
54+
ScalariformKeys.preferences := FormattingPreferences().
55+
setPreference(AlignParameters, true).
56+
setPreference(CompactControlReadability, true).
57+
setPreference(DoubleIndentClassDeclaration, true)
58+
59+
resolvers ++= Seq(
60+
"spymemcached" at "http://files.couchbase.com/maven2/",
61+
"zeroc" at "http://www.zeroc.com:8081/nexus/service/local/repositories/releases/content/",
62+
"SPI" at "http://www.spimageworks.com/maven/"
63+
)
64+
65+
{
66+
val activemq_ver = "5.12.0"
67+
// This should match the version spring_boot is trying to pull in.
68+
val spring_ver = "4.2.8.RELEASE"
69+
val spring_boot_ver = "1.3.8.RELEASE"
70+
libraryDependencies ++= Seq(
71+
"com.google.code.gson" % "gson" % "2.3.1",
72+
"com.google.guava" % "guava" % "12.0.1",
73+
"com.sun.mail" % "mailapi" % "1.5.4",
74+
"commons-lang" % "commons-lang" % "2.6",
75+
"com.zeroc" % "ice" % "3.6.1",
76+
"org.apache.activemq" % "activemq-pool" % activemq_ver,
77+
"org.apache.velocity" % "velocity" % "1.7",
78+
"org.jdom" % "jdom" % "1.1.3",
79+
"org.springframework.boot" % "spring-boot-starter-jdbc" % spring_boot_ver exclude ("org.springframework.boot", "spring-boot-starter-logging"),
80+
"org.springframework.boot" % "spring-boot-starter-log4j" % spring_boot_ver,
81+
"org.springframework.boot" % "spring-boot-starter-tomcat" % spring_boot_ver % "provided",
82+
"org.springframework.boot" % "spring-boot-starter-web" % spring_boot_ver,
83+
"org.springframework" % "spring-context-support" % spring_ver,
84+
"org.springframework" % "spring-jms" % spring_ver,
85+
"org.quartz-scheduler" % "quartz" % "2.2.1" exclude ("c3p0", "c3p0"),
86+
// "javax.inject" % "javax.inject" % "1",
87+
// "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided",
88+
// "javax.mail" % "javax.mail-api" % "1.5.4",
89+
"junit" % "junit" % "4.12" % "test",
90+
// https://github.com/sbt/junit-interface
91+
"com.novocode" % "junit-interface" % "0.11" % "test",
92+
"org.springframework.boot" % "spring-boot-starter-test" % spring_boot_ver % "test"
93+
)
94+
}
95+
96+
// reduce the maximum number of errors shown by the Scala compiler
97+
maxErrors := 5
98+
99+
// otherwise we get port conflicts in the tests
100+
parallelExecution in Test := false
101+
102+
fork in Test := true
103+
104+
// increase warnings generated by the Scala compiler
105+
scalacOptions ++= Seq(
106+
"-Xlint",
107+
"-Ywarn-dead-code",
108+
"-deprecation",
109+
"-feature",
110+
"-unchecked")
111+
112+
javacOptions ++= Seq(
113+
"-Xlint:deprecation",
114+
"-Xlint:unchecked")
115+
116+
// ask ScalaTest for full stack traces
117+
// testOptions in Test += Tests.Argument("-oF")
118+
119+
// name the war file 'cuebot.war'
120+
artifactName := { (config: sbt.ScalaVersion, module: ModuleID, artifact: Artifact) =>
121+
artifact.name + "." + artifact.extension
122+
}
123+
124+
// Faster incremental compilation
125+
incOptions := incOptions.value.withNameHashing(true)
126+
127+
deployTask := {
128+
"fab hot_redeploy:build=no" !
129+
}
130+
131+
deployTask <<= deployTask.dependsOn(xerial.sbt.Pack.packArchiveTbz)

0 commit comments

Comments
 (0)