Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stop-cluster on spot workers not raising Unsupported operation error #100

Open
arkal opened this issue Feb 10, 2016 · 1 comment
Open

stop-cluster on spot workers not raising Unsupported operation error #100

arkal opened this issue Feb 10, 2016 · 1 comment

Comments

@arkal
Copy link
Contributor

arkal commented Feb 10, 2016

Running stop-cluster on the cluster yields

Arjuns-MacBook-Pro:~ arjun$ cgcloud stop-cluster toil
INFO: Using zone 'us-west-1a' and namespace '/aarao/'
INFO: Binding to instance ... 
INFO: ... bound to i-3f11f78a.
INFO: === Performing stop() on workers ===
Thread-1 INFO: Binding to instance ... 
Thread-2 INFO: Binding to instance ... 
Thread-3 INFO: Binding to instance ... 
Thread-6 INFO: Binding to instance ... 
Thread-7 INFO: Binding to instance ... 
Thread-8 INFO: Binding to instance ... 
Thread-10 INFO: Binding to instance ... 
Thread-5 INFO: Binding to instance ... 
Thread-9 INFO: Binding to instance ... 
Thread-4 INFO: Binding to instance ... 
Thread-12 INFO: Binding to instance ... 
Thread-17 INFO: Binding to instance ... 
Thread-16 INFO: Binding to instance ... 
Thread-13 INFO: Binding to instance ... 
Thread-14 INFO: Binding to instance ... 
Thread-19 INFO: Binding to instance ... 
Thread-21 INFO: Binding to instance ... 
Thread-15 INFO: Binding to instance ... 
Thread-22 INFO: Binding to instance ... 
Thread-20 INFO: Binding to instance ... 
Thread-11 INFO: Binding to instance ... 
Thread-25 INFO: Binding to instance ... 
Thread-18 INFO: Binding to instance ... 
Thread-26 INFO: Binding to instance ... 
Thread-23 INFO: Binding to instance ... 
Thread-24 INFO: Binding to instance ... 
Thread-28 INFO: Binding to instance ... 
Thread-27 INFO: Binding to instance ... 
Thread-31 INFO: Binding to instance ... 
Thread-30 INFO: Binding to instance ... 
Thread-32 INFO: Binding to instance ... 
Thread-29 INFO: Binding to instance ... 
Thread-1 INFO: ... bound to i-4f3ee38c.
Thread-1 INFO: Stopping instance ...
Thread-2 INFO: ... bound to i-7a39e4b9.
Thread-6 INFO: ... bound to i-4e3ee38d.
Thread-13 INFO: ... bound to i-8339e440.
Thread-11 INFO: ... bound to i-b439e477.
Thread-10 INFO: ... bound to i-4d3ee38e.
Thread-5 INFO: ... bound to i-483ee38b.
Thread-24 INFO: ... bound to i-7f39e4bc.
Thread-31 INFO: ... bound to i-293ee3ea.
Thread-30 INFO: ... bound to i-b539e476.
Thread-14 INFO: ... bound to i-7839e4bb.
Thread-2 INFO: Stopping instance ...
Thread-6 INFO: Stopping instance ...
Thread-16 INFO: ... bound to i-7e39e4bd.
Thread-21 INFO: ... bound to i-8c39e44f.
Thread-19 INFO: ... bound to i-8939e44a.
Thread-3 INFO: ... bound to i-b139e472.
Thread-13 INFO: Stopping instance ...
Thread-25 INFO: ... bound to i-8e39e44d.
Thread-7 INFO: ... bound to i-283ee3eb.
Thread-11 INFO: Stopping instance ...
Thread-28 INFO: ... bound to i-b739e474.
Thread-23 INFO: ... bound to i-4c3ee38f.
Thread-5 INFO: Stopping instance ...
Thread-10 INFO: Stopping instance ...
Thread-22 INFO: ... bound to i-2b3ee3e8.
Thread-26 INFO: ... bound to i-7939e4ba.
Thread-17 INFO: ... bound to i-2f3ee3ec.
Thread-29 INFO: ... bound to i-8139e442.
Thread-31 INFO: Stopping instance ...
Thread-4 INFO: ... bound to i-7d39e4be.
Thread-12 INFO: ... bound to i-8839e44b.
Thread-20 INFO: ... bound to i-8f39e44c.
Thread-30 INFO: Stopping instance ...
Thread-14 INFO: Stopping instance ...
Thread-23 INFO: Stopping instance ...
Thread-3 INFO: Stopping instance ...
Thread-15 INFO: ... bound to i-8039e443.
Thread-18 INFO: ... bound to i-233ee3e0.
Thread-24 INFO: Stopping instance ...
Thread-8 INFO: ... bound to i-8d39e44e.
Thread-9 INFO: ... bound to i-2a3ee3e9.
Thread-27 INFO: ... bound to i-b639e475.
Thread-21 INFO: Stopping instance ...
Thread-16 INFO: Stopping instance ...
Thread-32 INFO: ... bound to i-b039e473.
Thread-19 INFO: Stopping instance ...
Thread-25 INFO: Stopping instance ...
Thread-28 INFO: Stopping instance ...
Thread-17 INFO: Stopping instance ...
Thread-26 INFO: Stopping instance ...
Thread-20 INFO: Stopping instance ...
Thread-7 INFO: Stopping instance ...
Thread-29 INFO: Stopping instance ...
Thread-22 INFO: Stopping instance ...
Thread-4 INFO: Stopping instance ...
Thread-12 INFO: Stopping instance ...
Thread-8 INFO: Stopping instance ...
Thread-15 INFO: Stopping instance ...
Thread-32 INFO: Stopping instance ...
Thread-27 INFO: Stopping instance ...
Thread-9 INFO: Stopping instance ...
Thread-18 INFO: Stopping instance ...
INFO: === Performing stop() on leader ===
INFO: Stopping instance ...
INFO: ... instance stopped.
Arjuns-MacBook-Pro:~ arjun$ cgcloud list toil-leader
INFO: Using zone 'us-west-1a' and namespace '/aarao/'
PI_LUSC aarao_toil-leader   0   172.31.10.233   None    i-3f11f78a  2016-02-10T07:32:50.000Z    stopped
Arjuns-MacBook-Pro:~ arjun$ cgcloud list toil-worker
INFO: Using zone 'us-west-1a' and namespace '/aarao/'
PI_LUSC aarao_toil-worker   0   172.31.20.143   54.183.159.51   i-4f3ee38c  2016-02-10T07:38:23.000Z    running
PI_LUSC aarao_toil-worker   1   172.31.26.160   54.183.75.197   i-7a39e4b9  2016-02-10T07:38:23.000Z    running
PI_LUSC aarao_toil-worker   2   172.31.26.187   54.67.25.118    i-b139e472  2016-02-10T07:38:23.000Z    running
PI_LUSC aarao_toil-worker   3   172.31.26.199   52.53.237.155   i-483ee38b  2016-02-10T07:38:23.000Z    running
PI_LUSC aarao_toil-worker   4   172.31.16.202   54.183.250.154  i-4e3ee38d  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   5   172.31.17.30    54.183.249.108  i-283ee3eb  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   6   172.31.17.40    54.193.17.232   i-8d39e44e  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   7   172.31.18.215   52.53.218.45    i-4d3ee38e  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   8   172.31.19.100   52.53.246.90    i-2a3ee3e9  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   9   172.31.19.11    54.193.26.220   i-7d39e4be  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   10  172.31.19.116   54.67.4.113 i-8839e44b  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   11  172.31.19.160   54.193.26.79    i-2f3ee3ec  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   12  172.31.20.176   54.183.217.21   i-8339e440  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   13  172.31.20.27    54.193.26.140   i-7e39e4bd  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   14  172.31.21.10    54.183.154.143  i-7839e4bb  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   15  172.31.21.158   54.183.194.171  i-8939e44a  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   16  172.31.21.204   54.183.191.27   i-8c39e44f  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   17  172.31.21.240   54.193.25.14    i-8039e443  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   18  172.31.22.85    54.183.194.28   i-2b3ee3e8  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   19  172.31.23.194   54.193.16.196   i-8f39e44c  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   20  172.31.23.85    54.183.174.172  i-b439e477  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   21  172.31.23.86    54.183.154.236  i-8e39e44d  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   22  172.31.24.227   54.193.17.250   i-233ee3e0  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   23  172.31.25.172   54.183.241.106  i-7939e4ba  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   24  172.31.26.110   54.67.92.202    i-4c3ee38f  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   25  172.31.26.98    54.193.25.191   i-7f39e4bc  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   26  172.31.27.220   54.183.168.157  i-b739e474  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   27  172.31.28.105   54.183.117.194  i-b639e475  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   28  172.31.28.127   54.193.16.204   i-293ee3ea  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   29  172.31.29.187   52.53.218.78    i-b539e476  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   30  172.31.29.251   52.53.253.106   i-8139e442  2016-02-10T07:38:24.000Z    running
PI_LUSC aarao_toil-worker   31  172.31.29.29    54.67.0.141 i-b039e473  2016-02-10T07:38:24.000Z    running

Attempting to manually stop an instance yields the correct error

INFO: Using zone 'us-west-1a' and namespace '/aarao/'
INFO: Binding to instance ... 
INFO: ... bound to i-b039e473.
INFO: Stopping instance ...
Traceback (most recent call last):
  File "/Users/arjun/my_bin/cgcloud", line 11, in <module>
    sys.exit(main())
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/cli.py", line 49, in main
    app.run( args )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/lib/util.py", line 300, in run
    command.run( options )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/commands.py", line 81, in run
    return self.run_in_ctx( options, ctx )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/commands.py", line 105, in run_in_ctx
    return self.run_on_role( options, ctx, role )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/commands.py", line 124, in run_on_role
    return self.run_on_box( options, box )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/commands.py", line 164, in run_on_box
    self.run_on_instance( options, box )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/commands.py", line 324, in run_on_instance
    getattr( box, self.name( ) )( )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/cgcloud/core/box.py", line 853, in stop
    self.ctx.ec2.stop_instances( [ instance.id ] )
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/boto/ec2/connection.py", line 1020, in stop_instances
    [('item', Instance)], verb='POST')
  File "/Users/arjun/cgcloud/venv/lib/python2.7/site-packages/boto/connection.py", line 1186, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>UnsupportedOperation</Code><Message>The instance 'i-b039e473' is a spot instance and may not be stopped.</Message></Error></Errors><RequestID>f28e5a4f-997c-4ace-9a84-b623de8d17a5</RequestID></Response>

FYI, cluster was created as

cgcloud create-cluster --leader-instance-type m3.medium --instance-type c3.8xlarge \
      --share /Users/arjun/LUSC_scaling/ --num-workers 32 --cluster-name PI_LUSC \
      --spot-bid 1.912 --leader-on-demand \
      --ssh-opts "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" toil
@arkal arkal changed the title stop-cluster not stopping workers stop-cluster o spot workers not raising error Feb 10, 2016
@arkal arkal changed the title stop-cluster o spot workers not raising error stop-cluster o spot workers not raising Unsupported operation error Feb 10, 2016
@arkal arkal changed the title stop-cluster o spot workers not raising Unsupported operation error stop-cluster on spot workers not raising Unsupported operation error Feb 10, 2016
@hannes-ucsc
Copy link
Contributor

Spot instances can't be stopped. I wonder, though, why you did't get any exceptions informing you about that fact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hannes-ucsc @arkal