01:07:19 advil: another issue you might run into; I think 3.2.2 added an option restricting the serving of static files to a specified base directory, so you may get webtiles games frozen at the splash screen with lots of 403 errors in the console 01:08:12 !source game_data_handler.py:13 01:08:13 https://github.com/crawl/crawl/blob/master/crawl-ref/source/webserver/game_data_handler.py#L13 01:09:22 that sets the base directory to the cwd, which is /crawl-master/webserver when launched via the init.d/webtiles script, not /crawl-master/ as it should be, since game data is stored next to the webserver dir 03:10:31 Windows builds of master branch on crawl.develz.org updated to: 0.25-a0-598-gc9c9e4cfe1 03:11:53 Unstable branch on crawl.beRotato.org updated to: 0.25-a0-598-gc9c9e4c (34) 03:48:35 Monster database of master branch on crawl.develz.org updated to: 0.24-a0-443-g80245de385 05:29:11 grats advil -- going to try py3 + tornado6 at some point this week 09:18:49 yeah, I still haven't restarted CAO yet, I keep feeling bad because whenever I check there's a ton of people playing 09:20:17 I think tornado 6 does need python 3 09:20:21 tornado 5 works with either 09:21:36 wow, thanks for pointing out the tornado version issue aidanh 09:21:56 it's running tornado 2.3 09:23:07 aidanh what docs are you referring to (server setup docs)? Do you think other servers have this? 09:23:20 I had no idea anyone was using something this old 09:30:21 it looks like I can't install tornado 5 on cao without updating the python version a bit, which I'm not excited to do 09:31:10 I've not tested the changes on tornado 4 because I didn't think that would be an issue...so 3.2.2 it is I guess 09:54:31 (that is, the changes are tested on 3.2.2 + py2, 5 + py2, 5 + py6, and 6 + py6) 10:06:38 advil: Can webcrawl server admins send a message to players? (Like a pending restart of the server) 10:06:51 no, we really need that feature 10:07:22 I actually started yesterday trying to figure out the best way to do it 10:07:32 I was remembering the "mail demon" that appeared for nethack, for purposes like that. 10:08:11 yeah nothing in crawl that does that, yet 10:08:34 these servers almost never restart, is probably why 10:09:05 most updates to crawl itself, or to the javascript side of the app, have an effect without restarting the server 10:12:38 aidanh: I can't figure out the static path thing you are alluding to, can you point me somewhere in the tornado docs? 10:17:58 that is, I can't figure out any reason from the docs to expect the problem I think you're suggesting will happen 10:25:12 aidanh you have been working on some kind of containerized server thing right? Is that usable by others yet? 10:34:40 aidanh ok I think I *should* have replicated the issue by running the server from the in-repo `webserver` directory instead of `source`, and in tornado 6 at least it worked fine 10:35:42 advil: re server setup docs, that would be https://crawl.develz.org/wiki/doku.php?id=dcss:server:setting_up_dgamelaunch_and_webtiles 10:36:00 it does seem weird that that class uses cwd 10:36:57 huh, I hope that doesn't mean that all servers are using this version 10:37:05 do you know what the patched version does? 10:44:04 ugh why does it always have to be so hard 10:45:28 re static path thing, https://www.tornadoweb.org/en/branch4.5/web.html#tornado.web.StaticFileHandler.validate_absolute_path 10:45:33 the `root` path that refers to is the path passed to initialize() in game_data_handler.py 10:46:44 the patched version adds a connection_timeout setting 10:49:28 yeah, I looked at the code for that function (in tornado 6 at least) 10:49:28 ah maybe testing it in webserver wouldn't do it 10:49:30 because game_data is still under there 10:50:17 so I guess the stupid fix would be to use `/` instead of `.` 10:50:53 that would be a perfectly reasonable interim fix 10:51:23 Unstable branch on crawl.akrasiac.org updated to: 0.25-a0-598-gc9c9e4c (34) 10:51:32 it seems like there should be a config option that is the path to where all the game data will be, but I'm not immediately seeing any config option on cao that would do that 10:51:42 ah, bad news 10:52:15 connection_timeout (i.e. idle_connection_timeout) was added in tornado 4 10:52:43 is this param something added by particular server owners? 10:53:13 it's in config.py as connection_timeout 10:53:35 set to 600 (seconds) by default, or you can specify None to disable it (which will work on tornado 3.2.2) 10:53:45 I see 10:54:52 I have no idea what the security ramifications of disabling it are though 10:54:52 maybe actually http_connection_timeout 10:54:52 in config.py 10:55:03 er, yes, sorry 10:55:04 `connection_timeout` in the default config.py seems not to ever be used 10:55:39 maybe I should revert 10:55:53 because this is a whole bunch of mess that I hadn't anticipated 10:56:48 it's set to 600 everywhere in the dgamelaunch-config repo 10:56:53 and for this particular one I'm not seeing an easy solution 10:57:13 (which has seen considerable drift from the mainline crawl config.py) 10:57:52 i don't think a revert is necessary, as long as the servers don't restart 10:58:20 yep, setting that value definitely does not work with tornado 3 10:58:33 I wonder if there's a monkeypatch fix I can come up with 10:59:33 please don't :) layers of fixes is how we got into this mess, after all 11:00:48 well, I'm not seeing any option that's backwards compatible with tornado 3 11:01:08 let alone 2 11:04:27 03Aidan Holm02 07* 0.25-a0-599-g2bf96b3: Fix broken static files base path 10(4 minutes ago, 1 file, 1+ 1-) 13https://github.com/crawl/crawl/commit/2bf96b3bd929 11:04:56 okay, that's that fixed 11:07:31 advil: oh, i should also mention that you'll need the same version of tornado installed inside the chroot 11:07:53 otherwise you will get module-not-found when tornado lazy loads after it's been chrooted 11:08:53 definitely not how it's currently set up on cao, there's nothing python related in the chroot 11:10:10 wrt the http_connection_timeout parameter, how about setting that config option to None based on tornado.version? 11:10:30 another fix would be to wrap the idle_connection_timeout setting in a try block and log a warning; no idea how much this would mess with servers 11:10:33 or else a try-except around the bit that initializes the server 11:10:45 yeah 11:10:50 I've mostly handled other stuff along these lines (less sever?) with try blocks, so maybe that 11:11:00 that's how the blocking call timeout stuff is currently handled 11:11:29 re your earlier question about containerized server setup, no, that was alexjurkiewicz who mentioned that 11:11:42 although I'd definitely be interested in it 11:12:00 ah ok, I got the impression that you had been testing in a containerized way for the chroot upgrade stuff you've been working on 11:12:01 btw no need to stay up for this if that's what you're doing :-) 11:12:50 no worries, had a bit too much coffee earlier ;) 11:13:20 not really, that was mostly focused on reproducing the current state of servers 11:13:37 apart from that, so far I've just been cleaning the shambling horror that is dgamelaunch-config 11:17:36 Unstable branch on crawl.kelbi.org updated to: 0.25-a0-599-g2bf96b3bd9 (34) 11:28:23 btw re the chroot stuff, it's basically ready to rumble, but iirc i was going to test that old binaries worked in the new chroot 11:28:43 otherwise it'd basically mean a forced upgrade of everyone's saves 11:34:03 03advil02 07* 0.25-a0-600-g9186476: Log an error on `idle_connection_timeout` for old Tornado versions 10(3 minutes ago, 1 file, 23+ 3-) 13https://github.com/crawl/crawl/commit/918647650c82 11:35:01 I still have no idea what the practical effect of ignoring that parameter will be, potentially idle connections just won't time out? 11:36:38 currently I'm going to try out cao on tornado 4 I think 11:37:11 would be ideal to do tornado 5, but you need the absolute lates py2.7 to do that 11:40:16 it would mean increased risk of running out of fds, i suppose 11:40:54 oh hm since cao doesn't use ssl, I wonder if tornado 5 would work 11:41:21 oh yeah I was going to ask about that 11:41:49 adding ssl would be a big improvement, but it's yet another fiddly thing to do 11:43:04 even with tornado 4 on cao, we'll still want the patch to server.py in case other servers restart for whatever reason 11:47:18 not sure what to do about the possibility that basically all servers are running the edlothiel patched version 11:47:29 Unstable branch on crawl.kelbi.org updated to: 0.25-a0-600-g918647650c (34) 11:47:37 and I think they must be, given this particular issue 11:48:48 oh, right, I had forgotten about the IOLoop.current issue 11:53:54 my goal was full backwards compatibility to ease the transition, but on the face of it I don't see how backwards compatibility to 2.4 is possible 11:55:13 what's the actual difference between IOLoop.instance() and IOLoop.current()? 11:55:58 that seems reasonably amenable to monkeypatching 11:55:58 hm yeah 11:56:26 IOLoop.current = staticmethod(IOLoop.instance) 11:56:28 ta da 11:57:54 yeah in fact in tornado 3 it has some caching but basically just calls instance 11:58:55 just tested, that works on tornado 2.4 with python 2.7.3 11:59:00 (in tornado 6 it's some incomprehensible asyncio stuff) 11:59:25 hm, I wonder what the difference is / intended use-cases are 11:59:28 heh, as always 12:00:26 I think part of this change was how having multiple ioloops was handled (now I think you're not supposed to do that at all) 12:01:22 https://www.tornadoweb.org/en/branch3.0/_modules/tornado/ioloop.html#IOLoop.current 12:01:43 if you have multiple ones then you can use `make_current` to switch which one is active in tornado 3 12:02:02 in 2 I think the user just had to keep track 12:02:59 oh, that makes sense 12:03:15 not a feature that you'd use in most situations, but very nice to have when you need it 12:21:43 ironically the "/" thing seems to have broken (only) tornado 2.4: GET /gamedata/164b5d7a90d1abbfaa01d841f60d5d20e2b0c772/icons.png (::1): /Users/advil/repos/crawl/crawl-ref/source/webserver/game_data/static/icons.png is not in root static directory 12:22:02 possibly only when running in the repo directory, not sure 12:22:59 it looks like there may be a few other tornado >=3 features I used too 12:25:08 Stable (0.23) branch on underhound.eu updated to: 0.23.1-93-ge536e68a2c 12:35:20 03advil02 07* 0.25-a0-601-g7348480: Tornado 2.4 compatibility hacks 10(2 minutes ago, 2 files, 35+ 3-) 13https://github.com/crawl/crawl/commit/734848013d98 12:36:39 03advil02 07* 0.25-a0-602-g631d167: checkwhite 10(16 seconds ago, 1 file, 1+ 1-) 13https://github.com/crawl/crawl/commit/631d16749a0c 12:38:26 aidanh sorry should have acknowledged you for the monkeypatch code, though maybe it's better not to be acknowledged for this ;-) 12:42:36 no worries, keep my name out of this :) 12:42:41 haha 12:43:09 ugh, yeah, I thought I tested that '/' thing on 2.4, but testing again reveals I clearly messed something up 12:43:21 it might have worked in a server configuration? 12:43:34 I didn't try to figure out what went wrong, just applied a brute force fix 12:44:47 https://www.tornadoweb.org/en/branch2.4/_modules/tornado/web.html , search for self.root 12:44:51 I'm pretty sure absolutely no one before today has used tornado 2.4 for local testing in the last 10 years 12:45:13 old versions of tornado add a /, and obviously nothing is going to be beneath // 12:45:38 oh heh 12:46:03 I noticed some weird compatibility code there when looking at more recent versions, maybe that is why 12:46:15 it seems to have been changed in 3.1 12:46:24 possibly just using "" would work? 12:46:43 probably the best workaround is to pass "" to initialize() but then immediately set self.root = '/' 12:47:43 Unstable branch on crawl.kelbi.org updated to: 0.25-a0-602-g631d16749a (34) 12:47:43 I think we might as well keep the brute force workaround 12:48:46 the one that checks tornado version? sure 12:48:48 since that'll exactly match how it was behaving for tornado 2.4, and it's easier to remove later 12:49:05 btw that version checking is going to go down in flames when it hits a version string like '3.2.2', no? 12:49:37 it's int(tornado.version[0]) 12:49:56 ah, right 12:50:38 looks like there's a useless try-except-raise in ensure_tornado_current()? 12:50:59 oh I guess tornado has ints in tornado.version_info 12:51:40 Unstable branch on crawl.akrasiac.org updated to: 0.25-a0-602-g631d167 (34) 12:53:08 oh, yeah 12:58:48 03advil02 07* 0.25-a0-603-g0321be9: A few cleanups to tornado compatibility code (aidanh) 10(51 seconds ago, 2 files, 16+ 22-) 13https://github.com/crawl/crawl/commit/0321be95db03 13:02:40 Unstable branch on crawl.kelbi.org updated to: 0.25-a0-603-g0321be95db (34) 13:06:23 Unstable branch on crawl.akrasiac.org updated to: 0.25-a0-603-g0321be9 (34) 13:19:58 The build was broken. (master - 7348480 #12833 : advil): https://travis-ci.org/crawl/crawl/builds/662755030 14:04:14 lol cao doesn't even have setuptools installed 14:27:12 ok, I think I can get CAO up to tornado 4 14:27:19 5 will be impossible without a python upgrade 14:32:00 currently planning doing the webtiles restart at 11pm EDT today 15:37:37 -!- The topic of ##crawl-dev is: Crawl Development | Logs: http://s-z.org/crawl-dev/ | People with +v have commit access. | Please keep general Crawl-related chat to ##crawl. | Dev wiki: http://crawl.develz.org/wiki | Long stuff to a pastebin service, please. 15:37:51 -!- The topic of ##crawl is: Play Dungeon Crawl Stone Soup online now! Type ??servers for instructions. | http://crawl.develz.org | FooTV: http://termcast.develz.org - ??footv for instructions | See also ##crawl-offtopic 17:08:04 man cao is back up to 97% 17:08:18 I guess I only bought us a few months 17:11:57 advil: aidanh and I were talking about how the dgl scripts leave crawl binaries in crawl-build/crawl-git-repository/.../source since they copy to install them instead of moving them 17:12:27 yeah, I noticed that 17:13:08 I may have done something about that already? 17:13:08 probably should have taken notes 17:13:21 they still exist as of now on CAO, just checked 17:13:28 but it's not all that much space, in the end 17:13:38 well, I didn't do anything that would prevent it from happening for new builds 17:14:25 if I'm remembering this one correctly, I did clean those up and at the time it was significant because it was all builds ever? 17:14:32 gotcha 19:23:17 Unstable branch on underhound.eu updated to: 0.25-a0-603-g0321be95db (34) 20:23:17 advil: anything I should be aware of as I go about making zig ghost levels a thing in terms of limits of ghost file loading? 20:23:25 !source ziggurat.lua:398 20:23:26 https://github.com/crawl/crawl/blob/master/crawl-ref/source/dat/dlua/ziggurat.lua#L398 20:23:42 this pan lord set will be my reference, so the weight of player ghosts will increase with zigs completed 20:23:49 and I'm mixing in other appropriate things 20:24:10 I guess testing it with zigs completed at 13 will tell me 20:29:11 you might want to look at what zig bones files are like in practice 20:29:29 my memory is they don't tend to be big 20:29:51 this is where saving ghosts by level is probably a bad strategy, would be more ideal to pull them in by xl or something 20:40:12 yeah, I will take a look at the zig ghost file 21:22:58 03advil02 07* 0.25-a0-604-g75b0bbc: Fix another bytes/string issue in a callback 10(15 minutes ago, 1 file, 2+ 2-) 13https://github.com/crawl/crawl/commit/75b0bbc335e9 21:22:58 03advil02 07* 0.25-a0-605-g7c741bd: Add a rudimentary serverwide announce facility to webtiles 10(4 minutes ago, 9 files, 185+ 7-) 13https://github.com/crawl/crawl/commit/7c741bda0139 21:24:16 03advil02 07* 0.25-a0-606-g85570c7: checkwhite new file 10(19 seconds ago, 1 file, 1+ 2-) 13https://github.com/crawl/crawl/commit/85570c75f3cd 21:25:10 this time I actually ran checkwhite, just not after doing git add 21:27:40 03advil02 07* 0.25-a0-607-geab1ddb: Remove some debug code 10(18 seconds ago, 1 file, 0+ 4-) 13https://github.com/crawl/crawl/commit/eab1ddba54e8 21:32:39 Unstable branch on crawl.kelbi.org updated to: 0.25-a0-607-geab1ddba54 (34) 21:38:47 can't wait to abuse my new admin panel 21:39:00 server-wide message: STOP LEARNING NECROMUTATION AND LEAVING DEATH'S DOOR AND BORGS UNMEMORIZED!!! 21:41:25 I wonder if I should identify the admin who made the announcement in the message :-P 21:43:04 does anyone know under what circumstances DGLACCT_PASSWD_LOCK and DGLACCT_EMAIL_LOCK are intended to be used? 21:52:04 are you seeing those from the dgamelaunch source? perhaps just to have a shared public account 21:53:39 ah I see 21:53:43 yes, they are in the dgl source 21:54:30 https://github.com/paxed/dgamelaunch/blob/master/dgamelaunch.h#L39 21:55:01 or ours, https://github.com/crawl/dgamelaunch/blob/master/dgamelaunch.h#L43 21:56:33 is there an easy way to set those flags for account from the command line? 21:57:01 login lock would have been a better approach than what I've done for the users I banned 21:57:10 perhaps webtiles doesn't respect those flags though 21:57:30 it doesn't 21:57:48 floraline has a PR for login lock, that I'll look at soon 22:06:00 The build was broken. (master - 7c741bd #12836 : advil): https://travis-ci.org/crawl/crawl/builds/662877905 22:47:37 Unstable branch on crawl.akrasiac.org updated to: 0.25-a0-607-geab1ddb (34) 22:56:34 03advil02 07* 0.25-a0-608-ge46d086: Move admin panel fully into client.html 10(2 minutes ago, 2 files, 18+ 17-) 13https://github.com/crawl/crawl/commit/e46d08612470 22:59:56 Unstable branch on crawl.akrasiac.org updated to: 0.25-a0-608-ge46d086 (34) 23:02:30 Unstable branch on crawl.kelbi.org updated to: 0.25-a0-608-ge46d086124 (34) 23:03:16 Webtiles server stopped. 23:03:25 Webtiles server started. 23:04:06 Webtiles server stopped. 23:04:25 Webtiles server started. 23:05:20 http://crawl.akrasiac.org:8080 appears to be offline 23:05:44 svendre: it's being restarted 23:06:01 aidanh: thanks for the info :) 23:08:08 Webtiles server started. 23:09:13 well, disappointingly it just silently failed to work on tornado 4 in the installed configuration for no apparent reason at all 23:09:19 (I had tested it from my home directory) 23:09:26 it's back up using tornado 2.4 for now 23:19:43 It's acting funky for me, tried a few reloads -- hanging at the "loading..." screen. 23:20:46 advil, did you check the log file for errors? the logging is a mess, but exceptions usually go there 23:21:23 it looks like I'm getting 403 HTML codes on /gamedata pages 23:21:47 sounds like the static path issue 23:21:50 some kind of permissions issue (403 forbidden) 23:22:36 I'll try clearing out cookies and cache and get back to the login page 23:26:25 aidanh no errors 23:27:14 by failed to work, do you mean failed to start? 23:27:54 it appeared to start according to the log, but quit immediately and silently (I think) 23:27:58 8080 did not get bound 23:28:02 did not help - no cache, no cookies, restarted browser. I think something borked my game I was in during the reboot.. I can get to the list of game versions but when I launch, the pages are all 403 (forbidden) up to game.js. I can give the exact URL if helpful, seems to contain what looks like a UUID 23:28:12 and yes, it is currently getting 403 errors 23:28:26 Webtiles server stopped. 23:28:40 To debug, you could set daemon to false, comment out the logfile option, set log level to debug, cd to the webserver dir, and run python ./server.py directly 23:29:18 svendre, thanks, no need, we know the ones you mean 23:29:23 er, do those steps as root 23:29:42 svendre btw this is planned maintenance, was announced on the webtiles lobby 23:31:08 all good here, I'm not complaining - just came in to give a heads up just in case, sorry I didn't see the maintenance posting time, I was just in a game that poofed. I see it's a server thing... I'm pretty decent at troubleshooting services and python, if I can be any help please let me know. 23:31:53 yeah, we need better ways of announcing these things 23:34:50 if it quits immediately and silently, it may be because something is already bound to 8080 23:35:19 try a netstat -tulpn 23:36:31 the classic tulpn combo 23:36:55 lsof -i :8080 also works 23:38:28 it would if CAO had lsof! 23:40:13 or: ss -ltp :P 23:41:58 it's not actually silent, it's just the except wasn't getting logged 23:42:32 v weird, it fails in tornado on `from tornado.platform.epoll import EPollIOLoop`, which works fine as a direct command in a python console 23:44:18 you're probably running into the chroot import issue I alluded to 23:44:34 maybe? 23:44:41 webtiles server doesn't run in the chroot afaik 23:44:51 oh? it does have a chroot option 23:44:58 You need to install the same version of tornado inside the chroot, or else add "import tornado.platform.epoll at the top of server.py" 23:45:31 yes it does run inside the chroot 23:45:31 ah maybe it does 23:45:31 yeah, it's an option, but I think we enable that on those dgl setups 23:46:07 yeah, we have chroot = "%%DGL_CHROOT%%" in config.py 23:46:51 I guess I don't understand why tornado 2.4 didn't have this issue too, I can't find it in the chroot 23:46:54 I'm planning on merging at least dgamelaunch-config into the main repo btw 23:47:09 I guess old tornado did less lazy loadinh 23:47:35 by main repo, you crawl/crawl? 23:47:56 or just make a repo at crawl/dgamelaunch-config; I thought you'd done that 23:48:26 ok, that fixed that issue with tornado 4; still getting 403 errors on both tornado versions though 23:49:15 also some really desperate player apparently found my test instance on :8081 23:49:20 hehe 23:49:36 <-wasn't me 23:50:00 * Psymania tries :8081 23:50:08 starting looking at the dgamelaunch config with chroot, ugh, looks like a headache - mostly nethack references 23:50:30 there are lots of warnings about getting the paths right, and matching though.. all over the place in that config file 23:51:07 oh advil has probably seen this 23:51:16 crawl-master/crawl-git-e46d086124/data/web/static/floor.png is not in root static directory 23:51:22 for the 403 error 23:52:25 floor.png, wall.png, feat.png, main.png, player.png, icons.png, gui.png, style.css, simplebar.ss and game.js are all the 403's 23:55:16 I don't know if this is any help at all, but I use a chroot'ed account as a landing point for scanned documents and sftp, and it's pretty fussy - the permissions on /home/account have to be root:root otherwise it breaks. 23:56:49 oh, actually, I'm getting all sorts of library errors unrelated to tornado too 23:58:22 ownership of the files inside the jail are owned by the account:account, so, I'm just talking about the actual directory in home for the account should be like chown /home/account root:root 23:58:58 there's a specific problem with how tornado restricts serving static files 23:59:03 leading to the 403 errors 23:59:13 aidanh and I thought we'd worked through it earlier but apparently not 23:59:45 I see.. 23:59:53 the traceback should show the file raising the error; I'd edit that file and add all the locals to the exception message