Wikimedia Tech Team (wikimediatech)

Wikimedia Tech Team

Tim: on kaulen: increased MaxClients to 500 to better deal with the connection flood

Wikimedia Tech Team

Tim: bugzilla is mostly working now, although it's very slow. The DDoS requests are blocked after connection setup using <Location>

Wikimedia Tech Team

Tim: on kaulen: restored MaxClients

Wikimedia Tech Team

logmsgbot: LocalisationUpdate completed (1.18) at Mon Feb 13 02:17:50 UTC 2012

Wikimedia Tech Team

Tim: temporarily moved bugzilla to port 444 until the connection flood (~1k req/s) subsides

Wikimedia Tech Team

Tim: started apache with MaxClients=30

Wikimedia Tech Team

Tim: after kaulen came back up, it was immediately overloaded with jsonrpc.cgi. Stopped apache.

Wikimedia Tech Team

Tim: kaulen is not responding on ssh, web down, rebooting

Wikimedia Tech Team

mark: Killed lsearchd processes on search8, restarted

Wikimedia Tech Team

mark: Rebalanced mw API app servers from load 120 to 150 in pybal list

Wikimedia Tech Team

mark: Increased MaxClients to 100 on API apaches in Puppet

Wikimedia Tech Team

mark: Restricted only opensearch API requests to the API squids

Wikimedia Tech Team

mark: Restricted only opensearch API requests to the API backend apaches, other API requests now hit the main mediawiki cluster

Wikimedia Tech Team

mark: maximum_forwards change deployed to all squids

Wikimedia Tech Team

mark: Set maximum_forwards 2 in squid.conf, deployed to the API squids only so far, rest is pending

Wikimedia Tech Team

binasher: restarted lsearchd on search{3,4,9}

Wikimedia Tech Team

logmsgbot: LocalisationUpdate completed (1.18) at Sun Feb 12 02:19:17 UTC 2012

Wikimedia Tech Team

apergos: restarted lightty on dataset2

Wikimedia Tech Team

RobH: manual test of each affected service complete, db9 fully online.

Wikimedia Tech Team

RobH: db9 moved, all systems online