sensu-dashboard dies with "undefined method..."

[amosshapira]

I'm setting up a test Sensu server on a Ubuntu 12.04 vagrant box (and Puppet 2.7) and keep getting the following in the logs once I hit the first page (which render with an "Oh no!" message and a red Bootstrap error pup-up at the bottom right corner of the page):

"timestamp":"2013-09-09T01:49:59.858940+0000","level":"info","message":"GET /all","remote_address":"33.33.33.1","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.65 Safari/537.36","request_method":"GET","request_uri":"/all?_=1378691404350","request_body":""}
/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:210:in `block (3 levels) in <class:Server>': undefined method `response' for nil:NilClass (NoMethodError)
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:209:in `each'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:209:in `detect'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:209:in `block (2 levels) in <class:Server>'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:151:in `call'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:151:in `set_deferred_status'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:191:in `succeed'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/multi.rb:53:in `check_progress'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/multi.rb:42:in `block in add'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:158:in `call'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:158:in `set_deferred_status'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:198:in `fail'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/client.rb:123:in `on_error'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/client.rb:117:in `unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:168:in `block in unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:168:in `map'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:168:in `unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:31:in `unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:1438:in `event_callback'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:36:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/bin/sensu-dashboard:10:in `<top (required)>'
        from /opt/sensu/bin/sensu-dashboard:23:in `load'
        from /opt/sensu/bin/sensu-dashboard:23:in `<main>'

And of course the dashboard process dies, leaving the stale pid file behind.

Is this expected? How stable is it in general and can we rely on it for production purposes?

[gposton]

+1
sensu 10.2 on centOS

[skymob]

This is also happening for me on a Vagrant running the the Sensu stack at 0.10.2-1:

[root@sensu-server ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)
[root@sensu-server ~]# rpm -qa | grep sensu
sensu-0.10.2-1.x86_64

I've not seen this problem on my full testing or production deployments, which are both running Sensu 0.10.0-1.

[skymob]

Here's a gist of the full stacktrace and line before it from /var/log/sensu/sensu-dashboard.log: https://gist.github.com/skymob/6617807

[amosshapira]

BTW - I'm not sure which of the steps I made is the one which "did it" but in general it seems that once I got the authentication to RabbitMQ sorted out the dashboard came to life. On another instance we also got this problem even though the authentication seemed to have been done right and the only way we got around it was to nuke the Vagrant instance and start from a fresh Vagrant (+Puppet) install. Voodoo.

[skymob]

@amosshapira ah, thanks. I didn't even notice that RabbitMQ had died because I'd changed the Vagrant hostname manually. I destroyed and re-upped the Vagrant and now Dashboard is fine. Dashboard should probably be able to handle that situation though.

[amdprophet]

A fix for this is being worked on in Pull Request #152.

[amdprophet]

@amosshapira @skymob are either of you able to test Sensu Dashboard 0.10.4 to see if this issue is fixed?

[amosshapira]

@amdprophet sorry I don't have a ready environment to test it right now (changed jobs, wiped my laptop, went on holiday).

[amdprophet]

@amosshapira No problem, thanks for reporting the bug!