Use Google's project MediaPipe Face Mesh as FACS tracker

[NumesSanguis]

Overview

As @fire shared in issue 27, Google has released MediaPipe Face Mesh as a Python library (code).

As this solution seems to be fast and is Apache-2.0 Licensed, it's interesting to consider this as a replacement for OpenFace. The tracking output is 468 3D coordinates (x, y, z) for a single mask (1 image or 1 frame in a video), which looks like:

To be useful for this project, the changes in these masks need to be mapped to Action Unit (AU) values according to the Facial Action Coding System (FACS) model.
More info - Demonstration of facial movement per AU.

This can be tricky, because the deformation in the tracking mesh when 2 AUs are active at the same time, won't be always just be adding the deformation of the AUs individually (no linear relation between e.g. AU12+AU14 compared to AU12 and AU14 separately). In practice a linear approach might be good enough though.

A paper by OpenFace on how they did Facial Action Unit detection:

• Cross-dataset learning and person-specific normalisation for automatic Action Unit detection Tadas Baltrušaitis, Marwa Mahmoud, and Peter Robinson in Facial Expression Recognition and Analysis Challenge, IEEE International Conference on Automatic Face and Gesture Recognition, 2015

MediaPipe Face Mesh output

A single face mesh tracking a single person in 1 image/frame has 468*3 values. A single landmark:

x: 0.4966168701648712
y: 0.694128692150116
z: -0.11288105696439743

Note that this output is normalized between [-1, 1].

Full landmark output single mesh (CLICK ME)

landmark {
  x: 0.4966168701648712
  y: 0.694128692150116
  z: -0.11288105696439743
}
landmark {
  x: 0.4816158711910248
  y: 0.6072000861167908
  z: -0.21528172492980957
}
landmark {
  x: 0.48926183581352234
  y: 0.6384062170982361
  z: -0.1202792227268219
}
landmark {
  x: 0.4422526955604553
  y: 0.5044909715652466
  z: -0.15802232921123505
}
landmark {
  x: 0.47746509313583374
  y: 0.5734008550643921
  z: -0.22716723382472992
}
landmark {
  x: 0.47410061955451965
  y: 0.5310384035110474
  z: -0.2105686366558075
}
landmark {
  x: 0.46738678216934204
  y: 0.4304444193840027
  z: -0.10389776527881622
}
landmark {
  x: 0.21002808213233948
  y: 0.4479711651802063
  z: 0.06500452011823654
}
landmark {
  x: 0.45994189381599426
  y: 0.353225439786911
  z: -0.07487861067056656
}
landmark {
  x: 0.45476841926574707
  y: 0.3084910809993744
  z: -0.08058346062898636
}
landmark {
  x: 0.44055771827697754
  y: 0.14465117454528809
  z: -0.038242045789957047
}
landmark {
  x: 0.49868014454841614
  y: 0.7090015411376953
  z: -0.10894951224327087
}
landmark {
  x: 0.5007204413414001
  y: 0.719012439250946
  z: -0.09897053241729736
}
landmark {
  x: 0.5020402073860168
  y: 0.7213298082351685
  z: -0.08386010676622391
}
landmark {
  x: 0.5109937787055969
  y: 0.7895573973655701
  z: -0.05537683516740799
}
landmark {
  x: 0.5119189023971558
  y: 0.8027503490447998
  z: -0.06261561065912247
}
landmark {
  x: 0.5133177638053894
  y: 0.8185326457023621
  z: -0.0709986463189125
}
landmark {
  x: 0.5152280330657959
  y: 0.8369268774986267
  z: -0.06422298401594162
}
landmark {
  x: 0.5181258916854858
  y: 0.8634383678436279
  z: -0.03230631723999977
}
landmark {
  x: 0.48471203446388245
  y: 0.6251006722450256
  z: -0.19659900665283203
}
landmark {
  x: 0.4531421959400177
  y: 0.6249157190322876
  z: -0.143781840801239
}
landmark {
  x: 0.06447488069534302
  y: 0.339648574590683
  z: 0.25507593154907227
}
landmark {
  x: 0.31962549686431885
  y: 0.46791741251945496
  z: 0.02018388733267784
}
landmark {
  x: 0.28245049715042114
  y: 0.47231966257095337
  z: 0.025203807279467583
}
landmark {
  x: 0.24612677097320557
  y: 0.47224801778793335
  z: 0.0373702310025692
}
landmark {
  x: 0.19500768184661865
  y: 0.45980608463287354
  z: 0.07198791205883026
}
landmark {
  x: 0.349759966135025
  y: 0.45816195011138916
  z: 0.022274039685726166
}
landmark {
  x: 0.26011133193969727
  y: 0.389184832572937
  z: 0.010085818357765675
}
landmark {
  x: 0.2995329797267914
  y: 0.389138400554657
  z: 0.008145865984261036
}
landmark {
  x: 0.22314757108688354
  y: 0.39661189913749695
  z: 0.02493373677134514
}
landmark {
  x: 0.19949522614479065
  y: 0.40996548533439636
  z: 0.0419645681977272
}
landmark {
  x: 0.16591835021972656
  y: 0.48124581575393677
  z: 0.09229511767625809
}
landmark {
  x: 0.3812064528465271
  y: 0.9071664214134216
  z: 0.021440977230668068
}
landmark {
  x: 0.19616428017616272
  y: 0.4427569508552551
  z: 0.07880867272615433
}
landmark {
  x: 0.062373995780944824
  y: 0.48197007179260254
  z: 0.26922407746315
}
landmark {
  x: 0.12454494833946228
  y: 0.465819776058197
  z: 0.13364587724208832
}
landmark {
  x: 0.2905968725681305
  y: 0.5917161703109741
  z: -0.022236289456486702
}
landmark {
  x: 0.4381605386734009
  y: 0.6930720210075378
  z: -0.10508478432893753
}
landmark {
  x: 0.4468565285205841
  y: 0.7214252352714539
  z: -0.08921617269515991
}
landmark {
  x: 0.38093671202659607
  y: 0.7045804262161255
  z: -0.07571420818567276
}
landmark {
  x: 0.3459462523460388
  y: 0.717242956161499
  z: -0.033613234758377075
}
landmark {
  x: 0.4050546884536743
  y: 0.7251520156860352
  z: -0.06517799943685532
}
landmark {
  x: 0.37378859519958496
  y: 0.7300823926925659
  z: -0.02689925581216812
}
landmark {
  x: 0.3168252110481262
  y: 0.7854228019714355
  z: 0.044089797884225845
}
landmark {
  x: 0.45147567987442017
  y: 0.6086890697479248
  z: -0.21051371097564697
}
landmark {
  x: 0.4435341954231262
  y: 0.5764272809028625
  z: -0.22110529243946075
}
landmark {
  x: 0.13545703887939453
  y: 0.3788211941719055
  z: 0.056899067014455795
}
landmark {
  x: 0.3665546476840973
  y: 0.5107970833778381
  z: -0.027126528322696686
}
landmark {
  x: 0.36413857340812683
  y: 0.6033427715301514
  z: -0.11284682154655457
}
landmark {
  x: 0.36061158776283264
  y: 0.5862759947776794
  z: -0.10040289163589478
}
landmark {
  x: 0.1746581494808197
  y: 0.5945844054222107
  z: 0.03535617142915726
}
landmark {
  x: 0.4421435296535492
  y: 0.5369409322738647
  z: -0.1966618001461029
}
landmark {
  x: 0.2176908552646637
  y: 0.33583539724349976
  z: -0.014324278570711613
}
landmark {
  x: 0.16782251000404358
  y: 0.35119521617889404
  z: 0.01882544904947281
}
landmark {
  x: 0.09129291772842407
  y: 0.2760707437992096
  z: 0.179345965385437
}
landmark {
  x: 0.3814221918582916
  y: 0.3533117473125458
  z: -0.05903608724474907
}
landmark {
  x: 0.3350331783294678
  y: 0.3979620933532715
  z: 0.016856539994478226
}
landmark {
  x: 0.2812685966491699
  y: 0.745897650718689
  z: 0.04884406924247742
}
landmark {
  x: 0.1452232003211975
  y: 0.7861756086349487
  z: 0.3580620586872101
}
landmark {
  x: 0.394314706325531
  y: 0.619657576084137
  z: -0.10019056499004364
}
landmark {
  x: 0.4270922541618347
  y: 0.6279042959213257
  z: -0.10671922564506531
}
landmark {
  x: 0.307145357131958
  y: 0.7430539727210999
  z: 0.049138471484184265
}
landmark {
  x: 0.32793134450912476
  y: 0.7431167960166931
  z: 0.03924649953842163
}
landmark {
  x: 0.14394527673721313
  y: 0.3307373821735382
  z: 0.03859829157590866
}
landmark {
  x: 0.3629467487335205
  y: 0.6155763268470764
  z: -0.09240224212408066
}
landmark {
  x: 0.28702664375305176
  y: 0.3346884250640869
  z: -0.04128420352935791
}
landmark {
  x: 0.27642199397087097
  y: 0.30713507533073425
  z: -0.04866618663072586
}
landmark {
  x: 0.22702038288116455
  y: 0.1813761293888092
  z: 0.023917950689792633
}
landmark {
  x: 0.11776086688041687
  y: 0.3024034798145294
  z: 0.10050833225250244
}
landmark {
  x: 0.2545614540576935
  y: 0.24051368236541748
  z: -0.01349948812276125
}
landmark {
  x: 0.11170089244842529
  y: 0.36651477217674255
  z: 0.09082090854644775
}
landmark {
  x: 0.08806124329566956
  y: 0.35485026240348816
  z: 0.16924317181110382
}
landmark {
  x: 0.44142016768455505
  y: 0.71026611328125
  z: -0.10029156506061554
}
landmark {
  x: 0.39390215277671814
  y: 0.7174962162971497
  z: -0.07107415795326233
}
landmark {
  x: 0.35953131318092346
  y: 0.7237300276756287
  z: -0.032877374440431595
}
landmark {
  x: 0.40349727869033813
  y: 0.6242803335189819
  z: -0.09295294433832169
}
landmark {
  x: 0.3165252208709717
  y: 0.7428818941116333
  z: 0.043773844838142395
}
landmark {
  x: 0.3406851887702942
  y: 0.7669117450714111
  z: 0.023747634142637253
}
landmark {
  x: 0.33578944206237793
  y: 0.7433992028236389
  z: 0.03797805681824684
}
landmark {
  x: 0.4140060544013977
  y: 0.6110385656356812
  z: -0.15295974910259247
}
landmark {
  x: 0.38158589601516724
  y: 0.7267777323722839
  z: -0.022487040609121323
}
landmark {
  x: 0.41331616044044495
  y: 0.724240779876709
  z: -0.0519661121070385
}
landmark {
  x: 0.4527042806148529
  y: 0.7233290076255798
  z: -0.07439269125461578
}
landmark {
  x: 0.45655256509780884
  y: 0.8648825287818909
  z: -0.02545144222676754
}
landmark {
  x: 0.4548318684101105
  y: 0.8358229398727417
  z: -0.05548474192619324
}
landmark {
  x: 0.4551684558391571
  y: 0.8169925212860107
  z: -0.059868987649679184
}
landmark {
  x: 0.457076758146286
  y: 0.8008837699890137
  z: -0.05086479336023331
}
landmark {
  x: 0.4605860710144043
  y: 0.7895804643630981
  z: -0.0449383407831192
}
landmark {
  x: 0.38548389077186584
  y: 0.7749221324920654
  z: 9.880734432954341e-05
}
landmark {
  x: 0.3780013620853424
  y: 0.7787157893180847
  z: -0.005501376464962959
}
landmark {
  x: 0.36844146251678467
  y: 0.7868350148200989
  z: -0.009895708411931992
}
landmark {
  x: 0.3618044853210449
  y: 0.7972811460494995
  z: -0.0017165766330435872
}
landmark {
  x: 0.3194640278816223
  y: 0.6933881044387817
  z: -0.021864671260118484
}
landmark {
  x: 0.08811995387077332
  y: 0.6305686235427856
  z: 0.4192250669002533
}
landmark {
  x: 0.48704424500465393
  y: 0.6317288279533386
  z: -0.14750073850154877
}
landmark {
  x: 0.36362749338150024
  y: 0.76473468542099
  z: 0.025699688121676445
}
landmark {
  x: 0.35120195150375366
  y: 0.7645043134689331
  z: 0.02254905179142952
}
landmark {
  x: 0.4399850070476532
  y: 0.6405894756317139
  z: -0.10764143615961075
}
landmark {
  x: 0.375720739364624
  y: 0.6322762966156006
  z: -0.06489422917366028
}
landmark {
  x: 0.4340003728866577
  y: 0.634506344795227
  z: -0.10850734263658524
}
landmark {
  x: 0.32653310894966125
  y: 0.5300822854042053
  z: -0.012731536291539669
}
landmark {
  x: 0.2632318139076233
  y: 0.5526027083396912
  z: 0.00027940748259425163
}
landmark {
  x: 0.3542672097682953
  y: 0.5996798276901245
  z: -0.07833056151866913
}
landmark {
  x: 0.14197686314582825
  y: 0.2222341001033783
  z: 0.09827900677919388
}
landmark {
  x: 0.16972631216049194
  y: 0.26265406608581543
  z: 0.04034826159477234
}
landmark {
  x: 0.20060905814170837
  y: 0.3098430633544922
  z: -0.006581175606697798
}
landmark {
  x: 0.35248780250549316
  y: 0.8181872367858887
  z: 0.022957870736718178
}
landmark {
  x: 0.36348414421081543
  y: 0.3094140589237213
  z: -0.07173669338226318
}
landmark {
  x: 0.3437490463256836
  y: 0.23109906911849976
  z: -0.047013651579618454
}
landmark {
  x: 0.3235037922859192
  y: 0.15789145231246948
  z: -0.02233033999800682
}
landmark {
  x: 0.21419644355773926
  y: 0.4688110947608948
  z: 0.05552274361252785
}
landmark {
  x: 0.12787866592407227
  y: 0.5059182047843933
  z: 0.1152096688747406
}
landmark {
  x: 0.36812901496887207
  y: 0.44856852293014526
  z: 0.025466639548540115
}
landmark {
  x: 0.15955564379692078
  y: 0.42317861318588257
  z: 0.07820062339305878
}
landmark {
  x: 0.3913554549217224
  y: 0.4891124963760376
  z: -0.04222859442234039
}
landmark {
  x: 0.38729652762413025
  y: 0.5918189883232117
  z: -0.15105858445167542
}
landmark {
  x: 0.09255272150039673
  y: 0.5330958962440491
  z: 0.16015303134918213
}
landmark {
  x: 0.1535634994506836
  y: 0.5218596458435059
  z: 0.07908810675144196
}
landmark {
  x: 0.20109590888023376
  y: 0.5274339914321899
  z: 0.04150325804948807
}
landmark {
  x: 0.27242377400398254
  y: 0.5198872685432434
  z: 0.020176483318209648
}
landmark {
  x: 0.326151967048645
  y: 0.5059536695480347
  z: 0.008998933248221874
}
landmark {
  x: 0.36502182483673096
  y: 0.48930275440216064
  z: -0.0038086730055510998
}
landmark {
  x: 0.4342425763607025
  y: 0.44238021969795227
  z: -0.08627469837665558
}
landmark {
  x: 0.09740403294563293
  y: 0.5966350436210632
  z: 0.1479249894618988
}
landmark {
  x: 0.12877383828163147
  y: 0.42056652903556824
  z: 0.09765920042991638
}
landmark {
  x: 0.46656739711761475
  y: 0.6248356699943542
  z: -0.19346565008163452
}
landmark {
  x: 0.3698480725288391
  y: 0.5399408936500549
  z: -0.046744342893362045
}
landmark {
  x: 0.0581088662147522
  y: 0.4841393232345581
  z: 0.3943922519683838
}
landmark {
  x: 0.3915879726409912
  y: 0.47015851736068726
  z: -0.011566780507564545
}
landmark {
  x: 0.3505880832672119
  y: 0.6014127135276794
  z: -0.03828176483511925
}
landmark {
  x: 0.18235325813293457
  y: 0.44442564249038696
  z: 0.08718197047710419
}
landmark {
  x: 0.38435113430023193
  y: 0.5708659887313843
  z: -0.13555653393268585
}
landmark {
  x: 0.1089932918548584
  y: 0.7077124118804932
  z: 0.39826714992523193
}
landmark {
  x: 0.3669191300868988
  y: 0.4378426671028137
  z: 0.03585735708475113
}
landmark {
  x: 0.41278910636901855
  y: 0.5510349273681641
  z: -0.17305362224578857
}
landmark {
  x: 0.22341036796569824
  y: 0.8427634239196777
  z: 0.1659410148859024
}
landmark {
  x: 0.24016138911247253
  y: 0.8845473527908325
  z: 0.22497878968715668
}
landmark {
  x: 0.07366946339607239
  y: 0.6184796690940857
  z: 0.28494521975517273
}
landmark {
  x: 0.16882428526878357
  y: 0.8082045912742615
  z: 0.208804190158844
}
landmark {
  x: 0.07401168346405029
  y: 0.41425713896751404
  z: 0.23039989173412323
}
landmark {
  x: 0.38395026326179504
  y: 0.9384512305259705
  z: 0.038852691650390625
}
landmark {
  x: 0.4726627469062805
  y: 0.6314657926559448
  z: -0.14449647068977356
}
landmark {
  x: 0.3361490070819855
  y: 0.5637073516845703
  z: -0.03239408880472183
}
landmark {
  x: 0.09436336159706116
  y: 0.47562286257743835
  z: 0.1737174242734909
}
landmark {
  x: 0.24819177389144897
  y: 0.4541197419166565
  z: 0.03995387256145477
}
landmark {
  x: 0.28098762035369873
  y: 0.45491164922714233
  z: 0.02891157753765583
}
landmark {
  x: 0.33149421215057373
  y: 0.7714042663574219
  z: 0.0297850389033556
}
landmark {
  x: 0.1068907082080841
  y: 0.6629747748374939
  z: 0.15780605375766754
}
landmark {
  x: 0.4563775956630707
  y: 0.9773697257041931
  z: 0.038378626108169556
}
landmark {
  x: 0.34909868240356445
  y: 0.9424223899841309
  z: 0.12394849210977554
}
landmark {
  x: 0.30104225873947144
  y: 0.9184457063674927
  z: 0.1700422614812851
}
landmark {
  x: 0.4470309317111969
  y: 0.22389227151870728
  z: -0.061588384211063385
}
landmark {
  x: 0.5305874347686768
  y: 0.9793791770935059
  z: 0.02257702499628067
}
landmark {
  x: 0.3115778863430023
  y: 0.4516664147377014
  z: 0.02560044266283512
}
landmark {
  x: 0.34036505222320557
  y: 0.44540536403656006
  z: 0.029373547062277794
}
landmark {
  x: 0.35830408334732056
  y: 0.4412270188331604
  z: 0.03582789748907089
}
landmark {
  x: 0.10178148746490479
  y: 0.41643524169921875
  z: 0.13728414475917816
}
landmark {
  x: 0.3333684802055359
  y: 0.4224480390548706
  z: 0.023977747187018394
}
landmark {
  x: 0.30116382241249084
  y: 0.4180963933467865
  z: 0.019164299592375755
}
landmark {
  x: 0.2705670893192291
  y: 0.41848498582839966
  z: 0.022922012954950333
}
landmark {
  x: 0.23962080478668213
  y: 0.42437681555747986
  z: 0.03419533744454384
}
landmark {
  x: 0.21877902746200562
  y: 0.4323749542236328
  z: 0.047679878771305084
}
landmark {
  x: 0.0542866587638855
  y: 0.40489107370376587
  z: 0.33343178033828735
}
landmark {
  x: 0.2258961796760559
  y: 0.45137596130371094
  z: 0.0531965009868145
}
landmark {
  x: 0.492154061794281
  y: 0.6571369767189026
  z: -0.10802311450242996
}
landmark {
  x: 0.35693883895874023
  y: 0.6745429039001465
  z: -0.05522865429520607
}
landmark {
  x: 0.39345255494117737
  y: 0.6159301996231079
  z: -0.11440737545490265
}
landmark {
  x: 0.43464553356170654
  y: 0.6622862219810486
  z: -0.10015508532524109
}
landmark {
  x: 0.4644954800605774
  y: 0.3917056918144226
  z: -0.07517721503973007
}
landmark {
  x: 0.28190237283706665
  y: 0.879287838935852
  z: 0.1305902898311615
}
landmark {
  x: 0.33249735832214355
  y: 0.9081810712814331
  z: 0.08741878718137741
}
landmark {
  x: 0.44906002283096313
  y: 0.9580975770950317
  z: -0.003724485170096159
}
landmark {
  x: 0.1908087432384491
  y: 0.8453988432884216
  z: 0.29279109835624695
}
landmark {
  x: 0.35627108812332153
  y: 0.431122362613678
  z: 0.03127270191907883
}
landmark {
  x: 0.41472727060317993
  y: 0.49441081285476685
  z: -0.09768738597631454
}
landmark {
  x: 0.5268312692642212
  y: 0.9583680629730225
  z: -0.01725558005273342
}
landmark {
  x: 0.3982726037502289
  y: 0.9635181427001953
  z: 0.07418964058160782
}
landmark {
  x: 0.09216967225074768
  y: 0.6903148293495178
  z: 0.27550944685935974
}
landmark {
  x: 0.41915491223335266
  y: 0.7839378118515015
  z: -0.02422095090150833
}
landmark {
  x: 0.4124149680137634
  y: 0.7920427918434143
  z: -0.02981782704591751
}
landmark {
  x: 0.4065060317516327
  y: 0.8049683570861816
  z: -0.03447248786687851
}
landmark {
  x: 0.4030742049217224
  y: 0.8215083479881287
  z: -0.02789442613720894
}
landmark {
  x: 0.39689958095550537
  y: 0.8466787338256836
  z: -0.003043192205950618
}
landmark {
  x: 0.3447519540786743
  y: 0.7349370121955872
  z: 0.008834567852318287
}
landmark {
  x: 0.3311789631843567
  y: 0.7321010828018188
  z: 0.010857325978577137
}
landmark {
  x: 0.3195076584815979
  y: 0.7279363870620728
  z: 0.0107242651283741
}
landmark {
  x: 0.2904130518436432
  y: 0.7160662412643433
  z: 0.01643066480755806
}
landmark {
  x: 0.15070724487304688
  y: 0.6711384654045105
  z: 0.0781637653708458
}
landmark {
  x: 0.4137824773788452
  y: 0.4639121890068054
  z: -0.06148932874202728
}
landmark {
  x: 0.3897281885147095
  y: 0.4090869128704071
  z: 0.00622992729768157
}
landmark {
  x: 0.36975330114364624
  y: 0.41790977120399475
  z: 0.02349216863512993
}
landmark {
  x: 0.35541102290153503
  y: 0.7306694984436035
  z: 0.010990123264491558
}
landmark {
  x: 0.164760559797287
  y: 0.7537547945976257
  z: 0.13850508630275726
}
landmark {
  x: 0.41884663701057434
  y: 0.40231144428253174
  z: -0.046456996351480484
}
landmark {
  x: 0.38834986090660095
  y: 0.8745599389076233
  z: 0.010446273721754551
}
landmark {
  x: 0.4720120429992676
  y: 0.4969934821128845
  z: -0.17284521460533142
}
landmark {
  x: 0.43800702691078186
  y: 0.4754502773284912
  z: -0.125311017036438
}
landmark {
  x: 0.46979230642318726
  y: 0.4655134677886963
  z: -0.1363074779510498
}
landmark {
  x: 0.3925747275352478
  y: 0.544338583946228
  z: -0.09969377517700195
}
landmark {
  x: 0.5232637524604797
  y: 0.9283697009086609
  z: -0.03395295888185501
}
landmark {
  x: 0.5202719569206238
  y: 0.8925750255584717
  z: -0.03351389244198799
}
landmark {
  x: 0.45128118991851807
  y: 0.8941555619239807
  z: -0.02066991664469242
}
landmark {
  x: 0.2987120747566223
  y: 0.8035737872123718
  z: 0.06246912106871605
}
landmark {
  x: 0.32278764247894287
  y: 0.628929078578949
  z: -0.02323312498629093
}
landmark {
  x: 0.3410077691078186
  y: 0.8435471057891846
  z: 0.04044443741440773
}
landmark {
  x: 0.2297494113445282
  y: 0.6335470676422119
  z: -0.0015491320518776774
}
landmark {
  x: 0.28704535961151123
  y: 0.6627032160758972
  z: -0.005966142285615206
}
landmark {
  x: 0.2043558955192566
  y: 0.6869828104972839
  z: 0.03605557605624199
}
landmark {
  x: 0.44654372334480286
  y: 0.9296964406967163
  z: -0.01958140730857849
}
landmark {
  x: 0.3718560039997101
  y: 0.5627002120018005
  z: -0.07194902747869492
}
landmark {
  x: 0.2805273234844208
  y: 0.8365193605422974
  z: 0.09268306195735931
}
landmark {
  x: 0.3318604826927185
  y: 0.8726451396942139
  z: 0.06202562153339386
}
landmark {
  x: 0.25948265194892883
  y: 0.760127067565918
  z: 0.06235231086611748
}
landmark {
  x: 0.12568554282188416
  y: 0.7197843790054321
  z: 0.17079582810401917
}
landmark {
  x: 0.21877965331077576
  y: 0.7787196636199951
  z: 0.09545453637838364
}
landmark {
  x: 0.12219688296318054
  y: 0.7549031376838684
  z: 0.25807473063468933
}
landmark {
  x: 0.25807932019233704
  y: 0.7004751563072205
  z: 0.02069525420665741
}
landmark {
  x: 0.3937021791934967
  y: 0.5172567367553711
  z: -0.07004125416278839
}
landmark {
  x: 0.40121617913246155
  y: 0.6062255501747131
  z: -0.1600225418806076
}
landmark {
  x: 0.3769870102405548
  y: 0.6137251853942871
  z: -0.11848873645067215
}
landmark {
  x: 0.4142044186592102
  y: 0.5831300020217896
  z: -0.1871527135372162
}
landmark {
  x: 0.3571854829788208
  y: 0.3837273418903351
  z: -0.0017102970741689205
}
landmark {
  x: 0.2956382930278778
  y: 0.3691142201423645
  z: -0.003822853323072195
}
landmark {
  x: 0.2431805431842804
  y: 0.3689405918121338
  z: 0.0017890947638079524
}
landmark {
  x: 0.20002737641334534
  y: 0.3778613805770874
  z: 0.0207314882427454
}
landmark {
  x: 0.17119255661964417
  y: 0.39538031816482544
  z: 0.046379540115594864
}
landmark {
  x: 0.15769413113594055
  y: 0.4544641971588135
  z: 0.10427386313676834
}
landmark {
  x: 0.06413432955741882
  y: 0.5501613616943359
  z: 0.28021129965782166
}
landmark {
  x: 0.18775427341461182
  y: 0.49140042066574097
  z: 0.07144352793693542
}
landmark {
  x: 0.22720354795455933
  y: 0.49667030572891235
  z: 0.04704747349023819
}
landmark {
  x: 0.27706509828567505
  y: 0.49465706944465637
  z: 0.029038190841674805
}
landmark {
  x: 0.32410991191864014
  y: 0.48578399419784546
  z: 0.018875757232308388
}
landmark {
  x: 0.3597770035266876
  y: 0.4729805588722229
  z: 0.014622870832681656
}
landmark {
  x: 0.3826742172241211
  y: 0.4602789282798767
  z: 0.01001096609979868
}
landmark {
  x: 0.07308945059776306
  y: 0.5572210550308228
  z: 0.4202417731285095
}
landmark {
  x: 0.3781541585922241
  y: 0.6203401684761047
  z: -0.09991267323493958
}
landmark {
  x: 0.41601312160491943
  y: 0.5217124223709106
  z: -0.13004370033740997
}
landmark {
  x: 0.42697784304618835
  y: 0.60621577501297
  z: -0.19082331657409668
}
landmark {
  x: 0.44973915815353394
  y: 0.6207560896873474
  z: -0.17728112637996674
}
landmark {
  x: 0.42836445569992065
  y: 0.6117011904716492
  z: -0.17246049642562866
}
landmark {
  x: 0.3879000246524811
  y: 0.6280851364135742
  z: -0.08750980347394943
}
landmark {
  x: 0.45556584000587463
  y: 0.6235464215278625
  z: -0.18769052624702454
}
landmark {
  x: 0.4613252580165863
  y: 0.6296263337135315
  z: -0.14377787709236145
}
landmark {
  x: 0.3795766532421112
  y: 0.43754279613494873
  z: 0.0279754139482975
}
landmark {
  x: 0.39896324276924133
  y: 0.44095879793167114
  z: 0.003275311319157481
}
landmark {
  x: 0.4094507694244385
  y: 0.44369012117385864
  z: -0.02484704740345478
}
landmark {
  x: 0.2062455117702484
  y: 0.43818193674087524
  z: 0.06118197739124298
}
landmark {
  x: 0.18186673521995544
  y: 0.4249523878097534
  z: 0.06467242538928986
}
landmark {
  x: 0.5035853981971741
  y: 0.4999598562717438
  z: -0.16247594356536865
}
landmark {
  x: 0.7413772344589233
  y: 0.40681785345077515
  z: 0.03183317929506302
}
landmark {
  x: 0.5206395387649536
  y: 0.6203845143318176
  z: -0.14702844619750977
}
landmark {
  x: 0.8921987414360046
  y: 0.27469971776008606
  z: 0.20559513568878174
}
landmark {
  x: 0.6345242261886597
  y: 0.443726509809494
  z: -0.0012637971667572856
}
landmark {
  x: 0.6732664704322815
  y: 0.4422304034233093
  z: -0.0007405339856632054
}
landmark {
  x: 0.7107070684432983
  y: 0.4366280734539032
  z: 0.007454411592334509
}
landmark {
  x: 0.7611681222915649
  y: 0.41643911600112915
  z: 0.03679535910487175
}
landmark {
  x: 0.6014629602432251
  y: 0.4388699531555176
  z: 0.005170207004994154
}
landmark {
  x: 0.6754707098007202
  y: 0.35596051812171936
  z: -0.016539370641112328
}
landmark {
  x: 0.6357946395874023
  y: 0.362026184797287
  z: -0.012676878832280636
}
landmark {
  x: 0.7149510979652405
  y: 0.35779958963394165
  z: -0.005732965189963579
}
landmark {
  x: 0.7429794669151306
  y: 0.3676159083843231
  z: 0.00850668828934431
}
landmark {
  x: 0.798031747341156
  y: 0.4332960247993469
  z: 0.054415538907051086
}
landmark {
  x: 0.6666548848152161
  y: 0.8896511793136597
  z: 0.006440763361752033
}
landmark {
  x: 0.7551484107971191
  y: 0.39951521158218384
  z: 0.044434018433094025
}
landmark {
  x: 0.9230958223342896
  y: 0.41696596145629883
  z: 0.21836701035499573
}
landmark {
  x: 0.8410305976867676
  y: 0.41109052300453186
  z: 0.09126061201095581
}
landmark {
  x: 0.6870469450950623
  y: 0.5627361536026001
  z: -0.04433140903711319
}
landmark {
  x: 0.5558522939682007
  y: 0.6847144365310669
  z: -0.11138556152582169
}
landmark {
  x: 0.5563403964042664
  y: 0.7140500545501709
  z: -0.0948861837387085
}
landmark {
  x: 0.6192103028297424
  y: 0.688189685344696
  z: -0.08818458765745163
}
landmark {
  x: 0.6611127853393555
  y: 0.6955099701881409
  z: -0.05048443749547005
}
landmark {
  x: 0.6017453074455261
  y: 0.7119958996772766
  z: -0.07660623639822006
}
landmark {
  x: 0.6376242637634277
  y: 0.7120493650436401
  z: -0.04177974909543991
}
landmark {
  x: 0.7118440866470337
  y: 0.7589071989059448
  z: 0.023515647277235985
}
landmark {
  x: 0.5126743316650391
  y: 0.6043348908424377
  z: -0.21332086622714996
}
landmark {
  x: 0.5123692154884338
  y: 0.5711033940315247
  z: -0.22434364259243011
}
landmark {
  x: 0.8047051429748535
  y: 0.3262219727039337
  z: 0.017701640725135803
}
landmark {
  x: 0.5927931070327759
  y: 0.49367523193359375
  z: -0.041927769780159
}
landmark {
  x: 0.6082389950752258
  y: 0.5855504274368286
  z: -0.12623192369937897
}
landmark {
  x: 0.6092568635940552
  y: 0.5680200457572937
  z: -0.11419199407100677
}
landmark {
  x: 0.8092576861381531
  y: 0.5479861497879028
  z: -0.0014363480731844902
}
landmark {
  x: 0.507614016532898
  y: 0.5319874286651611
  z: -0.20096458494663239
}
landmark {
  x: 0.7031187415122986
  y: 0.2961500585079193
  z: -0.0406055711209774
}
landmark {
  x: 0.760782778263092
  y: 0.3038061857223511
  z: -0.014503950253129005
}
landmark {
  x: 0.8425694704055786
  y: 0.21568787097930908
  z: 0.13522790372371674
}
landmark {
  x: 0.5385988354682922
  y: 0.34032565355300903
  z: -0.06717018038034439
}
landmark {
  x: 0.6027284860610962
  y: 0.3763411343097687
  z: 0.00015584345965180546
}
landmark {
  x: 0.7395942211151123
  y: 0.7144449949264526
  z: 0.02430148981511593
}
landmark {
  x: 0.9136649370193481
  y: 0.7333142757415771
  z: 0.31204113364219666
}
landmark {
  x: 0.5830571055412292
  y: 0.6060899496078491
  z: -0.11036499589681625
}
landmark {
  x: 0.551142156124115
  y: 0.6191096901893616
  z: -0.11160721629858017
}
landmark {
  x: 0.7133134603500366
  y: 0.7155593037605286
  z: 0.027367359027266502
}
landmark {
  x: 0.6942542791366577
  y: 0.7182605862617493
  z: 0.019502172246575356
}
landmark {
  x: 0.7833856344223022
  y: 0.2792634069919586
  z: 0.0030102706514298916
}
landmark {
  x: 0.6151416301727295
  y: 0.5973612666130066
  z: -0.10532160103321075
}
landmark {
  x: 0.6298211812973022
  y: 0.30626943707466125
  z: -0.059158291667699814
}
landmark {
  x: 0.6340736746788025
  y: 0.27706363797187805
  z: -0.06689538061618805
}
landmark {
  x: 0.6665512323379517
  y: 0.14459574222564697
  z: -0.0008641325403004885
}
landmark {
  x: 0.8120237588882446
  y: 0.24646785855293274
  z: 0.06073474511504173
}
landmark {
  x: 0.6456632018089294
  y: 0.20777574181556702
  z: -0.03507254645228386
}
landmark {
  x: 0.830237865447998
  y: 0.3101172149181366
  z: 0.048747479915618896
}
landmark {
  x: 0.8616198301315308
  y: 0.29404959082603455
  z: 0.12381526082754135
}
landmark {
  x: 0.5572219491004944
  y: 0.702317476272583
  z: -0.10649286210536957
}
landmark {
  x: 0.6097401976585388
  y: 0.7025388479232788
  z: -0.08312451094388962
}
landmark {
  x: 0.6494945883750916
  y: 0.7038677930831909
  z: -0.04906775429844856
}
landmark {
  x: 0.5753908157348633
  y: 0.6119466423988342
  z: -0.10112249851226807
}
landmark {
  x: 0.7044703364372253
  y: 0.716620683670044
  z: 0.023401401937007904
}
landmark {
  x: 0.6838364601135254
  y: 0.7439297437667847
  z: 0.006240763235837221
}
landmark {
  x: 0.6874526739120483
  y: 0.7196060419082642
  z: 0.018882296979427338
}
landmark {
  x: 0.5566620826721191
  y: 0.6008176207542419
  z: -0.16197162866592407
}
landmark {
  x: 0.630964994430542
  y: 0.7098885178565979
  z: -0.036925382912158966
}
landmark {
  x: 0.5949345827102661
  y: 0.7120267152786255
  z: -0.062326859682798386
}
landmark {
  x: 0.5532914996147156
  y: 0.7164126634597778
  z: -0.08051052689552307
}
landmark {
  x: 0.5808660387992859
  y: 0.8571116924285889
  z: -0.031214017421007156
}
landmark {
  x: 0.5760382413864136
  y: 0.8280496001243591
  z: -0.06238376349210739
}
landmark {
  x: 0.571864902973175
  y: 0.8095350861549377
  z: -0.06722287088632584
}
landmark {
  x: 0.5670864582061768
  y: 0.7940603494644165
  z: -0.05826123431324959
}
landmark {
  x: 0.5617002248764038
  y: 0.7833285927772522
  z: -0.05180101841688156
}
landmark {
  x: 0.6394295692443848
  y: 0.7579297423362732
  z: -0.014121092855930328
}
landmark {
  x: 0.6471433639526367
  y: 0.7607876062393188
  z: -0.0198096614331007
}
landmark {
  x: 0.6577990651130676
  y: 0.7677105665206909
  z: -0.024386810138821602
}
landmark {
  x: 0.6662388443946838
  y: 0.77713942527771
  z: -0.01703520677983761
}
landmark {
  x: 0.681681215763092
  y: 0.6677284836769104
  z: -0.04070958122611046
}
landmark {
  x: 0.9427776336669922
  y: 0.568905770778656
  z: 0.3679347634315491
}
landmark {
  x: 0.6616540551185608
  y: 0.7447880506515503
  z: 0.00876859575510025
}
landmark {
  x: 0.6731821298599243
  y: 0.7429101467132568
  z: 0.005481828935444355
}
landmark {
  x: 0.5410251021385193
  y: 0.6335012912750244
  z: -0.11239670217037201
}
landmark {
  x: 0.6075263619422913
  y: 0.6156387329101562
  z: -0.07554959505796432
}
landmark {
  x: 0.5455762147903442
  y: 0.6267488598823547
  z: -0.11317762732505798
}
landmark {
  x: 0.6382776498794556
  y: 0.5067825317382812
  z: -0.03245118632912636
}
landmark {
  x: 0.7076281905174255
  y: 0.5195570588111877
  z: -0.026095494627952576
}
landmark {
  x: 0.6209862232208252
  y: 0.5803111791610718
  z: -0.09298138320446014
}
landmark {
  x: 0.7698853611946106
  y: 0.17035594582557678
  z: 0.06264714896678925
}
landmark {
  x: 0.7429320812225342
  y: 0.2153264582157135
  z: 0.008848011493682861
}
landmark {
  x: 0.7155844569206238
  y: 0.26735252141952515
  z: -0.034542590379714966
}
landmark {
  x: 0.6803761720657349
  y: 0.796616792678833
  z: 0.005564854014664888
}
landmark {
  x: 0.545661449432373
  y: 0.29406824707984924
  z: -0.0806029886007309
}
landmark {
  x: 0.5515839457511902
  y: 0.2136651575565338
  z: -0.05765911936759949
}
landmark {
  x: 0.5609226226806641
  y: 0.13781699538230896
  z: -0.034965626895427704
}
landmark {
  x: 0.7430508732795715
  y: 0.4281330108642578
  z: 0.022452041506767273
}
landmark {
  x: 0.844697117805481
  y: 0.4517696499824524
  z: 0.07315035909414291
}
landmark {
  x: 0.5802938938140869
  y: 0.4321155548095703
  z: 0.010565190576016903
}
landmark {
  x: 0.7901830077171326
  y: 0.3747103214263916
  z: 0.04007326439023018
}
landmark {
  x: 0.5615787506103516
  y: 0.4760693609714508
  z: -0.053956516087055206
}
landmark {
  x: 0.5785065293312073
  y: 0.5778008699417114
  z: -0.16147805750370026
}
landmark {
  x: 0.8918441534042358
  y: 0.4735209345817566
  z: 0.11267603933811188
}
landmark {
  x: 0.8184541463851929
  y: 0.4718348979949951
  z: 0.03932137042284012
}
landmark {
  x: 0.768517255783081
  y: 0.4847654700279236
  z: 0.007178612053394318
}
landmark {
  x: 0.6929854154586792
  y: 0.48831745982170105
  z: -0.006274426821619272
}
landmark {
  x: 0.6349441409111023
  y: 0.48269224166870117
  z: -0.011449138633906841
}
landmark {
  x: 0.591370165348053
  y: 0.4720451235771179
  z: -0.01922423392534256
}
landmark {
  x: 0.5042835474014282
  y: 0.4369489550590515
  z: -0.09103569388389587
}
landmark {
  x: 0.8984358310699463
  y: 0.5377718210220337
  z: 0.10063814371824265
}
landmark {
  x: 0.8244988918304443
  y: 0.3667304515838623
  z: 0.05605992674827576
}
landmark {
  x: 0.5030688643455505
  y: 0.62248295545578
  z: -0.19541682302951813
}
landmark {
  x: 0.5940446257591248
  y: 0.5232166647911072
  z: -0.061043690890073776
}
landmark {
  x: 0.9397686123847961
  y: 0.41795897483825684
  z: 0.3411884307861328
}
landmark {
  x: 0.5593149065971375
  y: 0.4572816789150238
  z: -0.02309775911271572
}
landmark {
  x: 0.6279747486114502
  y: 0.5811028480529785
  z: -0.052767619490623474
}
landmark {
  x: 0.7707387208938599
  y: 0.39914965629577637
  z: 0.05126340687274933
}
landmark {
  x: 0.5788552165031433
  y: 0.5565616488456726
  z: -0.14665107429027557
}
landmark {
  x: 0.9362665414810181
  y: 0.6493462920188904
  z: 0.34945130348205566
}
landmark {
  x: 0.580301821231842
  y: 0.4213067293167114
  z: 0.020215287804603577
}
landmark {
  x: 0.5423721075057983
  y: 0.5413954854011536
  z: -0.18082544207572937
}
landmark {
  x: 0.827576756477356
  y: 0.8029569983482361
  z: 0.13183309137821198
}
landmark {
  x: 0.826129674911499
  y: 0.847174346446991
  z: 0.19045203924179077
}
landmark {
  x: 0.9401122331619263
  y: 0.5557646751403809
  z: 0.2341695874929428
}
landmark {
  x: 0.8798336982727051
  y: 0.7598730325698853
  z: 0.16860802471637726
}
landmark {
  x: 0.8934718370437622
  y: 0.35113173723220825
  z: 0.18171405792236328
}
landmark {
  x: 0.6714124083518982
  y: 0.9213674068450928
  z: 0.0226733461022377
}
landmark {
  x: 0.5012639164924622
  y: 0.6293689608573914
  z: -0.14582233130931854
}
landmark {
  x: 0.6347501277923584
  y: 0.5416210889816284
  z: -0.049733251333236694
}
landmark {
  x: 0.8785412311553955
  y: 0.41621002554893494
  z: 0.12761190533638
}
landmark {
  x: 0.7031115293502808
  y: 0.4189247786998749
  z: 0.01105754915624857
}
landmark {
  x: 0.6701067686080933
  y: 0.4246559143066406
  z: 0.003528991714119911
}
landmark {
  x: 0.693998396396637
  y: 0.747170627117157
  z: 0.011281153187155724
}
landmark {
  x: 0.904225766658783
  y: 0.6057406663894653
  z: 0.11217426508665085
}
landmark {
  x: 0.6056143641471863
  y: 0.968888521194458
  z: 0.029406843706965446
}
landmark {
  x: 0.7156542539596558
  y: 0.9208263158798218
  z: 0.10170795768499374
}
landmark {
  x: 0.7654732465744019
  y: 0.8901197910308838
  z: 0.14202842116355896
}
landmark {
  x: 0.6382739543914795
  y: 0.4263345003128052
  z: 0.003714349353685975
}
landmark {
  x: 0.608167827129364
  y: 0.42451217770576477
  z: 0.01025357935577631
}
landmark {
  x: 0.5894688963890076
  y: 0.4233325719833374
  z: 0.01901676133275032
}
landmark {
  x: 0.8561718463897705
  y: 0.35812079906463623
  z: 0.09244342148303986
}
landmark {
  x: 0.610427737236023
  y: 0.40064048767089844
  z: 0.004954040050506592
}
landmark {
  x: 0.6419333219528198
  y: 0.39145931601524353
  z: -0.0035158481914550066
}
landmark {
  x: 0.6723035573959351
  y: 0.38722050189971924
  z: -0.0033594227861613035
}
landmark {
  x: 0.7050137519836426
  y: 0.38829874992370605
  z: 0.004632687196135521
}
landmark {
  x: 0.7280253767967224
  y: 0.3929325044155121
  z: 0.0157832819968462
}
landmark {
  x: 0.922637403011322
  y: 0.3383057117462158
  z: 0.2819123864173889
}
landmark {
  x: 0.7254979610443115
  y: 0.41271597146987915
  z: 0.021820107474923134
}
landmark {
  x: 0.636895477771759
  y: 0.6547951698303223
  z: -0.06942592561244965
}
landmark {
  x: 0.5817331075668335
  y: 0.6023143529891968
  z: -0.12369497120380402
}
landmark {
  x: 0.5518733859062195
  y: 0.6538554430007935
  z: -0.10611211508512497
}
landmark {
  x: 0.7720415592193604
  y: 0.8480919003486633
  z: 0.10292784124612808
}
landmark {
  x: 0.7224955558776855
  y: 0.8843857049942017
  z: 0.06571002304553986
}
landmark {
  x: 0.6056313514709473
  y: 0.949116587638855
  z: -0.012368883937597275
}
landmark {
  x: 0.8745077848434448
  y: 0.7999281883239746
  z: 0.251703143119812
}
landmark {
  x: 0.5891892910003662
  y: 0.4131283760070801
  z: 0.013970219530165195
}
landmark {
  x: 0.5339846611022949
  y: 0.4853363633155823
  z: -0.10536916553974152
}
landmark {
  x: 0.6645001769065857
  y: 0.9482040405273438
  z: 0.05757834389805794
}
landmark {
  x: 0.9371163845062256
  y: 0.6303572654724121
  z: 0.22724252939224243
}
landmark {
  x: 0.6042358875274658
  y: 0.7720387578010559
  z: -0.035778649151325226
}
landmark {
  x: 0.612248957157135
  y: 0.7790498733520508
  z: -0.04143768548965454
}
landmark {
  x: 0.6202538013458252
  y: 0.7911738753318787
  z: -0.04681113734841347
}
landmark {
  x: 0.6267709732055664
  y: 0.8070510029792786
  z: -0.040287576615810394
}
landmark {
  x: 0.6387888193130493
  y: 0.831239640712738
  z: -0.016214247792959213
}
landmark {
  x: 0.6726184487342834
  y: 0.7126936912536621
  z: -0.00914271455258131
}
landmark {
  x: 0.6843034029006958
  y: 0.7081231474876404
  z: -0.008467559702694416
}
landmark {
  x: 0.6945748329162598
  y: 0.7022805213928223
  z: -0.009355273097753525
}
landmark {
  x: 0.7200194597244263
  y: 0.6861039400100708
  z: -0.006583949085325003
}
landmark {
  x: 0.8542413711547852
  y: 0.6205570101737976
  z: 0.038892485201358795
}
landmark {
  x: 0.5316597819328308
  y: 0.45482513308525085
  z: -0.06983059644699097
}
landmark {
  x: 0.5485436916351318
  y: 0.3964008688926697
  z: -0.0035277754068374634
}
landmark {
  x: 0.5719646215438843
  y: 0.40180742740631104
  z: 0.009555423632264137
}
landmark {
  x: 0.6626864671707153
  y: 0.7099243998527527
  z: -0.0063500674441456795
}
landmark {
  x: 0.8646278977394104
  y: 0.7047960758209229
  z: 0.100004643201828
}
landmark {
  x: 0.5146359801292419
  y: 0.39463502168655396
  z: -0.0523100420832634
}
landmark {
  x: 0.6529819369316101
  y: 0.8578700423240662
  z: -0.0028281020931899548
}
landmark {
  x: 0.5042141675949097
  y: 0.4702150225639343
  z: -0.129316508769989
}
landmark {
  x: 0.5673672556877136
  y: 0.5313471555709839
  z: -0.10971097648143768
}
landmark {
  x: 0.5904504060745239
  y: 0.885657787322998
  z: -0.0272047258913517
}
landmark {
  x: 0.7341678738594055
  y: 0.7744194865226746
  z: 0.03884495422244072
}
landmark {
  x: 0.6638760566711426
  y: 0.6042520999908447
  z: -0.04157112166285515
}
landmark {
  x: 0.6976938247680664
  y: 0.8204447031021118
  z: 0.02156316302716732
}
landmark {
  x: 0.7590912580490112
  y: 0.5951480865478516
  z: -0.02994963340461254
}
landmark {
  x: 0.7086312174797058
  y: 0.6325660347938538
  z: -0.028385769575834274
}
landmark {
  x: 0.8003910779953003
  y: 0.6445255279541016
  z: 0.004138569813221693
}
landmark {
  x: 0.6013050675392151
  y: 0.9202969074249268
  z: -0.027641113847494125
}
landmark {
  x: 0.5956445336341858
  y: 0.5460949540138245
  z: -0.08496230840682983
}
landmark {
  x: 0.7609846591949463
  y: 0.8049650192260742
  z: 0.06619337201118469
}
landmark {
  x: 0.713670015335083
  y: 0.8484628796577454
  z: 0.04197729751467705
}
landmark {
  x: 0.7646187543869019
  y: 0.7252383828163147
  z: 0.034109391272068024
}
landmark {
  x: 0.899861216545105
  y: 0.6650470495223999
  z: 0.1269931197166443
}
landmark {
  x: 0.811811089515686
  y: 0.737910807132721
  z: 0.0630282610654831
}
landmark {
  x: 0.9194583296775818
  y: 0.6992863416671753
  z: 0.2123991996049881
}
landmark {
  x: 0.7495314478874207
  y: 0.6656218767166138
  z: -0.005609575193375349
}
landmark {
  x: 0.5628350973129272
  y: 0.5045331716537476
  z: -0.0812133252620697
}
landmark {
  x: 0.5678873062133789
  y: 0.5942215919494629
  z: -0.16940903663635254
}
landmark {
  x: 0.5977365374565125
  y: 0.597644031047821
  z: -0.12983109056949615
}
landmark {
  x: 0.5464522242546082
  y: 0.5735775828361511
  z: -0.194813534617424
}
landmark {
  x: 0.5745513439178467
  y: 0.36621636152267456
  z: -0.015208940021693707
}
landmark {
  x: 0.6327033042907715
  y: 0.341604083776474
  z: -0.023593256250023842
}
landmark {
  x: 0.6859936118125916
  y: 0.3331960439682007
  z: -0.023968541994690895
}
landmark {
  x: 0.7332467436790466
  y: 0.33573848009109497
  z: -0.011009084060788155
}
landmark {
  x: 0.7689438462257385
  y: 0.348848432302475
  z: 0.011183288879692554
}
landmark {
  x: 0.8013812899589539
  y: 0.40506184101104736
  z: 0.06539134681224823
}
landmark {
  x: 0.9359502792358398
  y: 0.48567837476730347
  z: 0.22824189066886902
}
landmark {
  x: 0.7763141393661499
  y: 0.44680675864219666
  z: 0.035724811255931854
}
landmark {
  x: 0.7362672686576843
  y: 0.4579707980155945
  z: 0.01578447036445141
}
landmark {
  x: 0.6839812994003296
  y: 0.4636895954608917
  z: 0.002962634898722172
}
landmark {
  x: 0.6337649822235107
  y: 0.46220487356185913
  z: -0.001766429515555501
}
landmark {
  x: 0.5947312116622925
  y: 0.45500612258911133
  z: -0.0008954411023296416
}
landmark {
  x: 0.5681023001670837
  y: 0.4459196925163269
  z: -0.00256124185398221
}
landmark {
  x: 0.9421491622924805
  y: 0.4931836724281311
  z: 0.3675062358379364
}
landmark {
  x: 0.5998513698577881
  y: 0.6042654514312744
  z: -0.11095136404037476
}
landmark {
  x: 0.5358161330223083
  y: 0.5127532482147217
  z: -0.13812489807605743
}
landmark {
  x: 0.5388860106468201
  y: 0.5981953740119934
  z: -0.19681857526302338
}
landmark {
  x: 0.5211951732635498
  y: 0.6159141659736633
  z: -0.1806589514017105
}
landmark {
  x: 0.5407022833824158
  y: 0.6038531064987183
  z: -0.17878368496894836
}
landmark {
  x: 0.5928868651390076
  y: 0.6134583950042725
  z: -0.09697843343019485
}
landmark {
  x: 0.5145755410194397
  y: 0.6193801164627075
  z: -0.190590038895607
}
landmark {
  x: 0.5130746960639954
  y: 0.6262079477310181
  z: -0.14627952873706818
}
landmark {
  x: 0.5662755370140076
  y: 0.42279717326164246
  z: 0.015079694800078869
}
landmark {
  x: 0.5467715263366699
  y: 0.4294167459011078
  z: -0.006547094322741032
}
landmark {
  x: 0.5349453687667847
  y: 0.4338741898536682
  z: -0.032735634595155716
}
landmark {
  x: 0.7428998947143555
  y: 0.39666956663131714
  z: 0.02821102924644947
}
landmark {
  x: 0.7656942009925842
  y: 0.37987834215164185
  z: 0.02971372753381729
}

Tracking caveats

As of v0.8.7 of Face Mesh, it seems they don't support asymmetric facial expressions (raising only 1 eyebrow or only the right corner of your lips).

Extra info

How to get the landmark coordinates out of the face mesh object: https://stackoverflow.com/questions/67141844/how-do-i-get-the-coordinates-of-face-mash-landmarks-in-mediapipe

Next steps

1. The creation of a mapping between the 468 3D coordinates (landmarks) output of Face Mesh to AU values (OpenFace uses: 1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 20, 23, 25, 26, 28, and 45).

This might be a research project on itself. Some thoughts:

• Manual approach: I imagine the cleanest data would be mask generated from a picture of a resting face and a mask generated from a picture with 1 AU at maximum value (maximum muscle contraction of that muscle group). Or a video of a person moving a single AU, but that seems hard to map by hand.
• Machine learning approach: Generate data by running this solution on videos annotated with AU values. Then creating a ML model that learns the mapping from landmarks to AU values.

[HarryXD2018]

Hi, I try the latest Mediapipe == v0.8.8 Face Mesh, it seems that the asymmetrical problem is fixed in the new version. I am working on the Mediapipe replacing OpenFace as well. @fire, @Hunanbean Please let me know if you have any progress. Thanks!

[fire]

Let's keep in contact. I'll write up what we have so far.

[fire]

Heres a sketch of my approach. I have tooling to bake any vertex cache animation to a skeleton anim, but we need to encode the mesh vertices from mediapipe back into fa units. I think theres is a way. Any suggestions? Like bake permutations of the mediapipe facial mesh from -1 to 1 and manually calculate a key on that track for each face unit

[HarryXD2018]

I am planning to train a neural network to predict each AU intensity or binary classification. Before that, couple preprocessing are required, like centralization and normalization, etc.

[NumesSanguis]

@HarryXD2018

asymmetrical problem is fixed in the new version

Awesome!

train a neural network to predict each AU intensity or binary classification.

That would be my suggested approach as well. I can't work on this full time, but I can lend you a hand if you would like to?

[HarryXD2018]

Thanks for your offer. I will let you know if I run into any problems.

[fire]

I woke up inspired. What do you think about this?

Given the tech from Facebook https://github.com/facebookresearch/dlrm or https://ngc.nvidia.com/catalog/resources/nvidia:dlrm_for_pytorch.

The input data is mediapipe animations and we label float values for facial units.

468 3D coordinates (landmarks) output of Face Mesh to AU values (1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 20, 23, 25, 26, 28, and 45).

So 468 mediapipe floats, 18 facial unit floats and zero categoric features and we label these animation values as true.

If we want, we can also add the 52 Perfect Sync floats and VRM facial blend shape floats. The input data would be pairwise values of the floats (mediapipe and perfect sync), (mediapipe and vrm), (media pipe and vrm), and (mediapipe and SRanipal).

• https://hinzka.hatenablog.com/entry/2020/08/15/145040

• https://virtualcast.jp/wiki/_media/%E3%83%A2%E3%83%87%E3%83%AB%E4%BD%9C%E6%88%90/vrm_blendshape2.png?cache=

A third tier would be SRanipal aka Vive facial trackers which is another large list.

https://booth.pm/ja/items/2475790

[fire]

I was researching ludwig-ai for automl approaches.

https://github.com/ludwig-ai/ludwig supports tabular, photo (video frames), timeseries and speech samples.

Are we able to collect some datasets with labelled data?

[uriafranko]

Hey @NumesSanguis , @HarryXD2018
Did you got any progress with getting AU from the face mesh solution?

[fire]

I was able to get a standalone data from mediapipe if anyone is still on this project.

https://github.com/V-Sekai/mediapipe

There's a binary in the github cicd actions for linux and windows.

[fire]

Can people provide sample datasets? I can try doing image -> tabular data-> tabular values mappings via ludwig.

[HarryXD2018]

@fire There are plenty public datasets provide AU annotation, like CK+ and DISFA (the dataset used in OpenFace).

[NumesSanguis]

Hi @uriafranko, just to manage expectations, I'm working on other things, so I don't have much time for this project. Others here are working on this topic.
I am interested in seeing this work out of pure interest, so I'm here to assist if anyone has any questions.

[ymohamed08]

Hello everyone, just wanted to express my interest and check what kind of help i can provide