Wikimedia Bugzilla is closed!

Mailled labs-l to find out who can generate the cert. http://lists.wikimedia.org/pipermail/labs-l/2013-May/001212.html

RT #5184

Moving out from RT to "Labs" > "Infrastructure", per Ryan :)

Did not block the creation of a login wiki bug 51622

But it does prevents us from using SUL2 on beta which is bug 51580

*** Bug 52121 has been marked as a duplicate of this bug. ***

We need beta labs to work for IE in default configurations for test purposes. See Bug 52121

The other browsers don't like it either. Why is IE so special to you?

in the configurations we're using, other browsers don't require an automated test to negotiate a warning screen, see the screen shot on Bug 52121

All browsers should force you to navigate that warning, unless you've somehow changed their configuration to somehow completely break that security feature.

Upping priority (a tiny bit) as we really need the beta cluster (hosted on labs) to be as similar to production as possible for automated testing. This is needed for that (specifically for SUL2, kinnnnnda important ;) ).

Alex: turning ssl certificate verification off when doing automated testing is reasonably common because lots of folks don't run with valid certs and don't have a good infrastructure for distributing their self signed certs.  That isn't to say it is a good practice, just a common one.

*** Bug 53113 has been marked as a duplicate of this bug. ***

*** Bug 49533 has been marked as a duplicate of this bug. ***

*** Bug 46621 has been marked as a duplicate of this bug. ***

Apparently the beta labs now forcefully redirect to https, which means all beta sides are unusable for me (Chrome) because there is no obvious way to accept the certificate for bits, so no resources are being loaded.

It seems to me that being able to test the new [[m:HTTPS]] stuff on beta is a rather high priority; otherwise beta should be fixed for its other uses by reverting the HTTPS redirect on it.

There's no interface with firefox, either. The workaround is to manually go to https://bits.beta.wmflabs.org/ and accept the certificate there.

We should probably disable default https on beta until we have the opportunity to fix this up properly.  The proper fix, from what I understand, is pretty complicated, and isn't something we can do with just a day or two of elbow grease.

For those playing along at home: The Beta Cluster currently does not have https by default turned on for this very reason (it broke browser tests).

Mostly it makes using beta labs manually a real hassle.

Is it not possible to disable certificate checking on the automated browser tests until we get a proper certificate? I remember this being possible years ago when I built a selenium cluster...

The automated tests were OK I think with maybe some small exceptions.  The hassle was having to visit all the hosts (e.g. bits) in your browser in order to use any beta wikis.

My original requests was to generate certificates for the beta wildcards domains using the 'Labs CA' certificate authority.  By adding that authority certificate in the browser (or accepting it), that should make the browser trust all the generated certs.

Ryan, it's possible to work around the problem, but the issue is that, for manual testing, it's too easy to blame the SSL problem for things that might be totally unrelated.  Let's deploy SSL either correctly or not-at-all to beta.

Understandable for manual testing. I just wanted to make sure the automated tests weren't actually blocked, since there's ways to relatively easily workaround the issue.

As I've mentioned in the past, I'm totally fine with getting an SSL cert for beta, assuming we ensure every project member in deployment-prep has a signed NDA.

In the future Yuvi's proxy will handle load balancing, HTTPS and such. We'll be able to drop the NDA requirement (and the SSL cert) once we have that.

(In reply to comment #25)
> In the future Yuvi's proxy will handle load balancing, HTTPS and such.
> We'll able to drop the NDA requirement (and the SSL cert) once we have that.

I dont think we should use  Yuvi proxy for beta since the point of the project is to closely reproduce production. Currently the SSL connections are handled by nginx proxies, whenever varnish supports it, they will land on the varnish frontends.

Ah. Indeed. Well, let's treat deployment-prep like tools and require NDAs for all roots...

(In reply to comment #25)
> As I've mentioned in the past, I'm totally fine with getting an SSL cert for
> beta, assuming we ensure every project member in deployment-prep has a signed
> NDA.
> 
> In the future Yuvi's proxy will handle load balancing, HTTPS and such. We'll
> be
> able to drop the NDA requirement (and the SSL cert) once we have that.

Not every deployment-prep member is capable of signing an NDA anyway.

There's also the option which had been discussed before: make a new labs project whose sole purpose would be to do SSL termination and would then relay to varnish.

Then *that* project would have the NDA req instead of the core beta project.

Hm. That isn't a bad idea. If we were to do that, I'd want to move the ssl terminators and the caching layer there, and strip private IP info at that layer. Then we can change beta's privacy policy to that of the projects.

Bumping this with a note that this affects our ability to run automated tests with IE10 only.  (Demo on request) Other versions of IE seem to be OK with the current situation on labs, but I haven't found a way around IE10's paranoia. 

It would be good to have some sort of solution for this in advance of support for IE10 in VisualEditor.

(In reply to comment #19)
> For those playing along at home: The Beta Cluster currently does not have
> https by default turned on for this very reason (it broke browser tests).

This is no longer the case.

(In reply to comment #24)
> Let's deploy SSL either correctly or not-at-all to beta.

Agreed.  I think turning SSL completely off on beta would be better than the current situation.  It's not good for people to get in the habit of ignoring certificate warnings.

Of course, fixing it properly with a valid cert is best.

(In reply to comment #32)
> (In reply to comment #19)
> > For those playing along at home: The Beta Cluster currently does not have
> > https by default turned on for this very reason (it broke browser tests).
> 
> This is no longer the case.

Correction: it's enabled by default, but only for logged in users.

But I can't find prefershttps in Preferences to turn that off.

Yes. Using beta continues to be a hassle, and in the near future the SSL issue is going to be a big problem.

(In reply to comment #33)
> (In reply to comment #32)
> > (In reply to comment #19)
> > > For those playing along at home: The Beta Cluster currently does not have
> > > https by default turned on for this very reason (it broke browser tests).
> > 
> > This is no longer the case.
> 
> Correction: it's enabled by default, but only for logged in users.
> 
> But I can't find prefershttps in Preferences to turn that off.

This means, for instance, that VisualEditor is entirely untestable on Beta Labs. Yay fun. :-(

(In reply to comment #35)
> (In reply to comment #33)
> > (In reply to comment #32)
> > > (In reply to comment #19)
> > > > For those playing along at home: The Beta Cluster currently does not have
> > > > https by default turned on for this very reason (it broke browser tests).
> > > 
> > > This is no longer the case.
> > 
> > Correction: it's enabled by default, but only for logged in users.
> > 
> > But I can't find prefershttps in Preferences to turn that off.
> 
> This means, for instance, that VisualEditor is entirely untestable on Beta
> Labs. Yay fun. :-(

That is not true, VisualEditor works just fine on beta labs, we found Bug 54791 on beta labs with an automated browser test just this week.  Beta labs is as healthy as it has ever been, and it is running the master branch with database updates all maintained automatically.  Varnish on beta is configured properly, and is working well. 

What does not work on beta labs without a real SSL cert are: 

* tests for IE10 are busted : https://saucelabs.com/jobs/a29bcd975ac6483195b3bd56a64fec6a

* having to visit https://bits.beta.wmflabs.org manually over and over again to get styling to work in every browser is a drag

Calm down, what we need is to use the 'Labs CA' certificate authority to generate all the certificates we need, then add that authority as a trusted one in the browsers.

I would do it myself if I had access to the 'Labs CA'.

Change 87045 had a related patch set uploaded by Mattflaschen:
Labs: Turn off secure login on loginwiki due to untrusted SSL

https://gerrit.wikimedia.org/r/87045

(In reply to comment #33)
> But I can't find prefershttps in Preferences to turn that off.

So it turns out it only displays if wgSecureLogin is on, but it defaults true either way.  The gotcha is that it wasn't not on for Labs, *except* loginwiki.  However, loginwiki is not taken into account by the Preferences logic.  

I believe the above should workaround that by disabling wgSecureLogin on loginwiki as well.  However, I haven't tested it, since I don't have a realistic CentralAuth environment.

This is just intended to be temporary until there's an SSL setup on Beta that is trusted out-of-the-box by browsers.

Has anyone considered creating a CA (http://pages.cs.wisc.edu/~zmiller/ca-howto/), generating a * cert, and adding that certificate to the browser's trust? Is that a possibility using the selenium service we're using?

I'm still not opposed to getting a proper cert, but that requires some infrastructure and permissions changes in deployment-prep...

(In reply to comment #29)
> There's also the option which had been discussed before: make a new labs
> project whose sole purpose would be to do SSL termination and would then
> relay
> to varnish.
> 
> Then *that* project would have the NDA req instead of the core beta project.

That's the dynamichttp proxy, which has been running silently for a while now :)

(In reply to comment #40)

> I'm still not opposed to getting a proper cert, but that requires some
> infrastructure and permissions changes in deployment-prep...

Since we've been going around on this since May, what would be the next step to make that happen?

(In reply to comment #41)
> (In reply to comment #29)
> > There's also the option which had been discussed before: make a new labs
> > project whose sole purpose would be to do SSL termination and would then
> > relay
> > to varnish.
> > 
> > Then *that* project would have the NDA req instead of the core beta project.
> 
> That's the dynamichttp proxy, which has been running silently for a while now
> :)

They want to have the same https infrastructure as production, which makes sense ;)

(In reply to comment #42)
> (In reply to comment #40)
> 
> > I'm still not opposed to getting a proper cert, but that requires some
> > infrastructure and permissions changes in deployment-prep...
> 
> Since we've been going around on this since May, what would be the next step
> to
> make that happen?

Maybe create another project with limited access and move the ssl terminators there. You should realize that other than advising I'm not really doing anything in beta...

Change 87045 merged by jenkins-bot:
Labs: Turn off secure login on loginwiki due to untrusted SSL

https://gerrit.wikimedia.org/r/87045

Summary of IRC conversation:

1. Remove projectadmin permissions from volunteers
2. Clean up sudo policies to disallow root on varnish systems (that will have real certs)
3. Buy * certs

Any volunteer that would like to have projectadmin in the project or root on the varnish systems will need an NDA.

please don't forget about wikidata.beta.wmflabs.org

It's going to be an all-inclusive * cert for beta.

Currently getting a cert warning when trying to go to:
https://deployment.wikimedia.beta.wmflabs.org/
This is breaking our automation tests for mobile.

(In reply to comment #49)
> Currently getting a cert warning when trying to go to:
> https://deployment.wikimedia.beta.wmflabs.org/
> This is breaking our automation tests for mobile.

Heh. Did you read the ticket any before posting this? :)

I guess the first part of my comment is redundant with what you already know. Mostly, I wanted to convey that our automation testing is broken, in case that affects prioritizing.

(In reply to comment #51)
> I guess the first part of my comment is redundant with what you already know.
> Mostly, I wanted to convey that our automation testing is broken, in case
> that
> affects prioritizing.

Is it possible for now to not use https? That's what other projects have done for now to workaround the issue.

Should be Real Soon Now: https://bugzilla.wikimedia.org/show_bug.cgi?id=55804

Any progress on this? It's been 2 weeks and we still can't do acceptance testing on beta labs.

And no we can't not use https:
http://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page
Although I have no idea why the HTTP version doesn't load.

(In reply to comment #54)
> Although I have no idea why the HTTP version doesn't load.

It does for me, both in browsers and using curl.  Is it possible you have HTTPS Everywhere or something enabled?

No I don't. It currently redirect to HTTPS for me in Safari and fails to load completely in Firefox.

This actually makes beta labs unusable for me and the rest of the mobile team. Mobile is dependent on https - it's been like this since login was first introduced. It is not easy to disentangle.

If MobileFrontend doesn't work without HTTPS it's broken. That's a different bug, though.

Anyway, this isn't blocked on ops. In comment 46 I listed the steps that need to be taken, and they can be taken by anyone that has projectadmin in deployment-prep. Someone needs to take ownership of this situation and run with it. It will take us a matter of days to get SSL certs and we can start that process now, but until the other steps are taken there's no major point in doing so.

Well, disabling secure login is easy - much better to have a couple of tests failing than not be able to run them at all.

Secure login is disabled

(In reply to comment #58)
> Anyway, this isn't blocked on ops. In comment 46 I listed the steps that need
> to be taken, and they can be taken by anyone that has projectadmin in
> deployment-prep. Someone needs to take ownership of this situation and run
> with it.

I think the problem is that no one is quite sure how to do those things. I'm not sure otherwise I would JFDI.

(In reply to comment #46)
> 2. Clean up sudo policies to disallow root on varnish systems (that will have
> real certs)

I assume this means that non-admins should not have sudo on these machines?

unassigning from Ryan to reflect reality.

So I took the bug thinking the cert was blocking everything, however that is not the case.

I will handle the purchase of the certificate, but I am not claiming any ownership over any other part of this process =]

We get purchase approvals within our closed ticketing system, RT #6116.  (I've set Hashar as requestor for that ticket.)  Once its purchased I'll post in here as well.

(In reply to comment #46)
> Summary of IRC conversation:
> 
> 1. Remove projectadmin permissions from volunteers

{{DONE}}, see list of members at: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep

> 2. Clean up sudo policies to disallow root on varnish systems (that will have
> real certs)

Not sure what needs to happen here. Tips/Pointers? I think Antoine had some ideas?

> 3. Buy * certs

In progress: https://rt.wikimedia.org/Ticket/Display.html?id=6116

> 1. Remove projectadmin permissions from volunteers

I also just removed TheDJ since he didn't have an NDA on file and he didn't respond to my email asking if he wanted to sign one.

> 2. Clean up sudo policies to disallow root on varnish systems (that will have
> real certs)

Apparently the sudo policies are set up at https://wikitech.wikimedia.org/wiki/Special:NovaSudoer. It looks like most of them have sudo enabled for "ALL" hosts. I imagine disabling their root privileges on varnish systems just entails unchecking some of these hosts. Unfortunately, I'm not sure which of these hosts are varnish systems. Is it all 4 of the deployment-cache hosts? Any others?

> 3. Buy * certs

Good to hear that's in progress.

I have cleaned up permissions on the deployment-prep labs project (ie: beta cluster).

The project admins are now limited to people from the Wikimedia ops and mw-core teams.

Root access has been limited to people having signed a non disclosure agreement with Wikimedia.


The reason for this change is to let us put real SSL certificates on the Varnish caches which would let us support HTTPS on the beta cluster.  We want to keep access to the certificates restricted, hence the change.


Please review the list of admins and sudo policy for the deployment-prep project on:

https://wikitech.wikimedia.org/wiki/Special:NovaProject
https://wikitech.wikimedia.org/wiki/Special:NovaSudoer

Buying certs is pending approval according to RobH a few days ago.  The related ticket is https://rt.wikimedia.org/Ticket/Display.html?id=6116

Reason in comment 16 is past (testing of new default HTTPS access), but warning message in Selenium probably still justifies highest prio? (for four months now)

(In reply to comment #68)
> The related ticket is https://rt.wikimedia.org/Ticket/Display.html?id=6116

To summarize the RT ticket: Prices offered by SSL vendors felt out of scope.

Greg and RobLa: RT ticket states you wanted to discuss how to proceed here.
Any updates?

Would it be an option to flatten our subdomains?

We'd only need beta.wmflabs.org and *.beta.wmflabs.org to be in the certificate

(at e.g. DigiCert, those wildcards are $1425 for 3 years includes root and *)

For example:

* beta.wmflabs.org
* bits.beta.wmflabs.org
* wikimedia.beta.wmflabs.org
* commons-wikimedia.beta.wmflabs.org
* wikipedia.beta.wmflabs.org
* en-wikipedia.beta.wmflabs.org
* nl-wikibooks.beta.wmflabs.org
* en-m-wikibooks.beta.wmflabs.org
* en-m-wikinews.beta.wmflabs.org

Greg and RobLa: RT ticket states you wanted to discuss how to proceed here.
Any updates (or should this not be highest priority)?

Still highest priority.  We want to get that done while I am in SF, hopefully this afternoon (PST time).

(In reply to comment #72)
> Still highest priority.  We want to get that done while I am in SF, hopefully
> this afternoon (PST time).

Sorry, was referring to another bug :-/


Regarding SSL certificates on beta, I am not sure what the status is.  Maybe Greg/Chris would know.  We might have a workaround now or simply stopped running browser tests over HTTPS.

We have stopped running browser tests over https. 

I think we still want SSL for labs, but I don't know of anyone actively working on that right now.

I think we do want it, on a limited set of subdomains to keep the cost down.

Beta's:
- loginwiki (so we can check SUL interactions)
- metawiki (so OAuth works correctly and securely)
- another wiki, maybe enwiki? (to catch other SUL login/wgSecureLogin issues)
- Oh, and to make enwiki work, the bits domain would also need a cert

Because of the necessity to have a default CA sign this, we need to buy individual certificates.  Please provide a definite list of domain names, and I'll get that ball rolling.

login.wikipedia.beta.wmflabs.org
meta.wikipedia.beta.wmflabs.org
en.wikipedia.beta.wmflabs.org
bits.beta.wmflabs.org
upload.beta.wmflabs.org (for some icons on meta/login)

That's all I can see from the network calls.

Antoine/Chris/Chris: confirm?

http://meta.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix is the full list.

There are some weird ones like 'ee-prototype'.

http://commons.wikimedia.beta.wmflabs.org/ is important

(In reply to Kunal Mehta (Legoktm) from comment #78)
> http://meta.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix is the full
> list.

For the avoidance of doubt: we're not doing them all, just a subset. SSL certs are a racket and expensive.

Yeah, we can't do all; we can't even reasonably all the necessary wildcards to cover the whole matrix.

I have six now; any more?

(In reply to Greg Grossmeier from comment #80)

> 
> For the avoidance of doubt: we're not doing them all, just a subset. SSL
> certs are a racket and expensive.

Oh, missed that above. Makes sense :)

wikidata.beta.wmflabs.org might be nice, since I know a few gadgets go cross domain to it.

I think the dewiki community also wanted to have de.wikipedia.beta.wmflabs.org, but unless we get a discount for buying them at one time, it would probably be good to wait until we have browser tests running against that wiki.

(Lowering priority and unassigning from self)

Status, afaict:
* Price was annoying, but we've identified the 7 we want (comment 77, comment 79, and comment 83).
* Setup was(is?) annoying because of the lack of easy way to secure these private certs from other non-WMF root labs users.

Marc/RobH: Let me know if I have that wrong and if there's anything else we can do right now.

(In reply to Greg Grossmeier from comment #84)
> * Setup was(is?) annoying because of the lack of easy way to secure these
> private certs from other non-WMF root labs users.

I thought comment 65 and comment 67 said the "access to private keys without an NDA" part was solved.

Antoine said he did the sudo configuration (see comment 67); I'm not sure if it's checked yet.

The beta cluster has for Varnish instances with a Nginx HTTPS proxy installed.  Nginx refuses to start because the star.wmflabs.org certificate is invalid:

root@deployment-cache-bits01:~# /etc/init.d/nginx start
Starting nginx: nginx: [emerg]
  SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key")
  failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
 nginx: configuration file /etc/nginx/nginx.conf test failed


To fix it we would need a few certificates to be installed on the instances via the role::protoproxy::ssl::beta puppet class in manifests/role/protoproxy.pp

star.wmflabs.org would cover the entries:

 bits.beta.wmflabs.org
 upload.beta.wmflabs.org
 wikidata.beta.wmflabs.org

We would need *.wikimedia.beta.wmflabs.org and *.wikipedia.beta.wmflabs.org certs as well.

https://gerrit.wikimedia.org/r/#/c/111386/
https://gerrit.wikimedia.org/r/#/c/126008/1

for *.wmflabs.org, the self-signed cert has recently been replaced with one from RapidSSL , at first the chained file, which is created by puppet was wrong, the above changes should have fixed that.

(In reply to Antoine "hashar" Musso from comment #86)
star.wmflabs.org would cover the entries:

 bits.beta.wmflabs.org
 upload.beta.wmflabs.org
 wikidata.beta.wmflabs.org

I'm afraid it can't and *.wmflabs.org is not *.beta.wmflabs.org (only one level of wildcard possible). But ask RobH to make sure.

 > I'm afraid it can't and *.wmflabs.org is not *.beta.wmflabs.org (only one
> level of wildcard possible). But ask RobH to make sure.

Ah indeed my bad. Sorry :-]

Won't Fix'ing this for now.

A) We have self-signed certs in place on Beta
B) Real certs are expensive
C) There hasn't been any team come with a specific use case where buying the real certs would make sense.

Feel free to reopen if any of the above changes.

This bug seems to be drifted away from the initial comment.
(In reply to Antoine "hashar" Musso from comment #0)
> We need certificates generated by 'Labs CA' for the entries listed in
> role::protoproxy::ssl::beta and some more.  I guess the easiest would be to
> create *.beta.wmflabs.org cert that will also contains the following DNS
> entries:

So

(In reply to Greg Grossmeier from comment #91)
> Won't Fix'ing this for now.
> A) We have self-signed certs in place on Beta

We still don't have valid self-signed certs for beta since eqiad migration as far as I know. At least nginx refuses to starts because of cert mismatch. As I was told in bug 63538 that this (certs) is handled here, I REOPEN. Please generate new (self-signed) certs.

I can think of three significant problems with self-signed certificates:

1. It trains people to ignore SSL warnings, which means they ignore them when it's a legit problem.
2. It causes problems with automated tests (not clear if all of these have been worked around).
3. It's a real pain when testing manually because you have to visit https://bits.beta.wmflabs.org/ (and maybe also upload.beta.wmflabs.org) manually in Firefox.

FWIW, https://en.wikipedia.beta.wmflabs.org isn't working at all (not even invalid) right now (can't connect), but I assume that's temporary

If cost is the issue, did we consider setting up our own certificate authority (chained to an existing root)?  It's an upfront cost, but as I understand it that means no per-cert cost.

Google and Microsoft both have them, just to name a couple.

(In reply to Matthew Flaschen from comment #93)
> If cost is the issue, did we consider setting up our own certificate
> authority (chained to an existing root)?  It's an upfront cost, but as I
> understand it that means no per-cert cost.
> 
> Google and Microsoft both have them, just to name a couple.

Getting a delegated signing certificate is a huge deal actually. Once you have one, any certificate you sign is trusted by all who trust your upstream signer. The x509 protocol does not make it possible to construct a trusted signing certificate that is restricted to a particular domain. As such, any trusted signer who issues delegate signing certificates must impose strict practices and regular audits of those practices on any delegate organization.