Moz Knowledge Base/Feature Requests/Moz Feature Requests

PlannedDoneNot planned

Open Site Explorer to crawl HTTPS

Epica
suggested this on December 16, 2011 08:01 AM

Campaign crawls for HTTPS is now working, anyway we can get this for Open Site Explorer?

 

Comments latest first

User photo
Miranda Rensch
SEOmoz

Yes! This is in testing right now and by the end of next week, a small amount of HTTPS sites will start being crawled in Open Site Explorer. Because of the nature of our index, it will take a few index updates for us to crawl and find a higher percentage of the HTTPS sites that exist because we have to find other sites linking to them in order to discover them.

Thanks everyone on this thread for your patience! This has been a much bigger task than we anticipated, and we're really excited to get it out for you all. 

Miranda Rensch - Product Manager

March 28, 2014 12:19 PM
User photo
iQuanti

Any update?

March 27, 2014 10:25 PM
User photo
Miranda Rensch
SEOmoz

Hi there! Our current ETA for https sites in OSE is four weeks from now. We plan to begin crawling https sites when the next index kicks off next week, and then if all goes well with the index it will be there 2-3 weeks after the index starts.

We apologize for the continued wait. This is something our team has been working on very hard for the last couple of months; thank you so much for your patience.

Miranda Rensch - Product Manager

March 03, 2014 05:38 PM
User photo
jaston1979

Is functionality with the crawlers turned on yet? Our OSE data is virtually useless right now.

March 03, 2014 01:59 PM
User photo
Carin Overturf
SEOmoz

Hey all,

Here is the latest status of where we are in regards to HTTPS sites being crawled and indexed in Open Site Explorer:

1. We're dev complete - huge step!

2. Friday the team walked through a really positive round of testing to make sure the the behaviors in Open Site Explorer were as expected

3. Next step: final testing and then turning on functionality in our crawlers.

 

I hate to keep saying "it's almost there", but we're really close at this point. As soon as we have the functionality turned on within our crawlers, I'll update this post and let you all know an ETA of when to expect to see HTTPS data within the index. Once the functionality is turned on within our crawlers, it will take about two to three weeks before it's processed in our index. The first index that does contain HTTPS data will be launched with a corresponding blog post as an announcement. 

I know you are all eager to see this functionality, as are we! I really appreciate your patience as we get closer to being ready for release!

 

Thanks,
Carin

February 24, 2014 11:37 AM
User photo
jsookman

Agreed... Any update? I've been waiting for this for almost a year already. As a paying customer with an HTTPS website, it's been pretty frustrating to say the least, not to have accurate data with which to make decisions.

February 21, 2014 08:44 AM
User photo
Mickaelwam

Hi Carin, any news on this?

February 21, 2014 08:37 AM
User photo
Carin Overturf
SEOmoz

Hey folks! Wanted to update this forum post with the latest information regarding including HTTPS data in Open Site Explorer. 

The code to include HTTPS websites in our index is complete, but, unfortunately, during testing, an elusive bug was discovered with pulling the HTTPS data into Open Site Explorer.
Our Big Data team is focused on implementing a fix and we hope to have this feature ready for Open Site Explorer in the first quarter of 2014. We're so close to finally having this highly requested feature ready for you all!
We'll definitely announce when this feature has been deployed to production, but I'll keep this thread updated as well. 
Thanks!!
Carin

 

January 15, 2014 02:06 PM
User photo
Carin Overturf
SEOmoz

Yep! We have just started testing of this feature and hope to start seeding HTTPS sites into our crawlers early November if things continue to go well. We'll only be seeding a few HTTPS sites initially, but as our crawlers organically discover and index more HTTPS sites, they will continually be added into the index.

Once we begin indexing HTTPS sites, you'll be able to see data across all of our apps if you site is in the in the index - including Moz Analytics, Open Site Explorer and the MozBar. We'll be publishing a blog post and updating this forum post as soon we are able to release HTTPS into the Mozscape index!

Thanks!

Carin

October 01, 2013 10:35 AM
User photo
Optimum4

Is this feature any closer to going live? Are you now at a point where you have a likely date it will be launched?

Thanks!

October 01, 2013 04:27 AM
User photo
Carin Overturf
SEOmoz

Hey Nic (and others subscribed to this thread)!

Looks like Adam has jumped in here to answer a few of the questions around adding HTTPS to Open Site Explorer, but let me fill you all in on the latest:

  1. HTTPS crawling is just about done with development and ready for testing! The initial code changes have been made in our Mozscape schedulers and crawlers, however, there are still a few bugs to work out so we're not actively crawling any HTTPS sites just yet.

    Our Big Data team is working on ironing out these last few issues and hope to have the bugs worked out soon so we can begin testing. 

  2. Once we do start including HTTPS sites into our crawlers, they will slowly propagate through - crawling for 38 days and then processing for 12 days.

    HTTPS page metrics will be included in the Mozscape index, which powers Open Site Explorer, along with all other link graph metrics. This means that on an index release day, you would see updated metrics for either an HTTP or HTTPS site in Open Site Explorer, Mozscape Public API, the MozBar, etc. If a secure page is listed in our SERP analysis tool and included in our Mozscape index, we will have data for that HTTPS page as well. 

  3. We'll be making a few changes in Open Site Explorer so it will be more clear the protocol of the site - http:// or https://. Stay tuned for these updates in the next few months!

I hope that helps answer your questions, but let me know if you have any others. I know this is a huge feature request from a lot of our users so we're really excited to get this one released as soon as we can. We're aggressively aiming for late September or early October and will be announcing the release on the Moz blog as soon as we start releasing HTTPS data.

Thanks,
Carin

 

 

 

July 26, 2013 04:24 PM
User photo
Nic Dale

Would like to know when this gets rolled out.  Frustrated being unable to use Moz to crawl my sites on https servers

July 25, 2013 01:16 PM
User photo
EconostoNL

Sorry, ignore my last post. Comments are sorted other way around, my bad.

So, let's hope to see those HTTPS stats soon, good work.

June 07, 2013 01:29 AM
User photo
EconostoNL

Any news in regards to this feature? Last response was about 18 months ago and I encountered the same issue as the people above as our entire website is HTTPS.

June 07, 2013 01:28 AM
User photo
kstoltz

Oh wow, this is fantastic and very timely. I'm working with a company that does lots of business with government agencies and a portion of its site is secure with a good deal of links to them, but it's tough getting a big picture of who is doing all that linking. Can't wait to see this.

May 16, 2013 04:55 AM
User photo
Adam Feldstein
SEOmoz

Hi Josh, 

As I understand it, the HTTPS crawl data will work it's way into the indexes iteratively over time, and this should start in one of the next few indexes. This means that we should have better and better secure site representation over the next few months.  I'll try and keep this thread posted as we see uptake with this.

With regard to the SERP analysis reports, if you are asking about metrics for https pages, this will come along for the ride, as will access via the API, campaigns, and OSE. 

I hope this helps. 

May 02, 2013 07:40 AM
User photo
jsookman

One more question, will HTTPS metrics also be enabled for SEOMoz PRO SERP analyses?

April 30, 2013 10:22 PM
User photo
jsookman

*meant "HTTPS option"

April 30, 2013 09:32 PM
User photo
jsookman

Hi Adam --

Does this mean OpenSiteExplorer will be supporting HTTPS websites (or an HTTS option) in its next crawl?

Thx,
Josh 

April 30, 2013 09:31 PM
User photo
Adam Feldstein
SEOmoz

Hi RKG-Dev -

When we release this you will be able to retrieve https URLs and associated metrics from the API in the same way you currently retrieve http metrics. We will be crawling https URLs at the same time we crawl http URLs so they will be available on the same dates.

April 26, 2013 01:50 PM
User photo
RKG-Dev

Will this be available in the API as well?  Will it be on the same dates?

April 26, 2013 12:02 PM
User photo
AtcoreDK

Really hoping for this feature too! 

April 04, 2013 04:16 AM
User photo
Tela Andrews
Moz

Hi Sophie -
We have the majority of the engineering work on this completed. I expect we'll be able to release this feature in the second quarter of this year.

Cheers,
Tela 

April 02, 2013 03:02 PM
User photo
sophiep

Hi,

I am eagerly waiting to work this feature in OSE. Can you please share some updates in this regards?

Sophie

March 18, 2013 02:15 AM
User photo
Miranda Rensch
SEOmoz

Hello there! Sorry about the wait. It turned out to be a more complicated change than we had expected originally. We just added the type of testing framework that we'll need to test this type of addition, so the addition should be in place by March of next year. Again, I appoligize for the wait. Thank you!

Miranda Rensch - Product Manager

December 04, 2012 01:24 PM
User photo
alexakaclarke

Any updates on this? It's been on the list and "a couple of months away" for years now.

December 04, 2012 01:32 AM
User photo
Adam Feldstein
SEOmoz

Sadly, this is still a ways out. This will require a significant change to Linkscape, and we are focused on some larger improvements to the scale and speed of the index first. Rest assured that this isn't forgotten and is still an important project in our backlog.

April 19, 2012 08:06 PM
User photo
Jed iCrossing

Any ways to get Open Sie Explorer to crawl HTTPS sites?

January 12, 2012 09:22 AM
User photo
Adam Feldstein
SEOmoz

Thanks for the suggestion!  This is on our list of important product updates, but is still at least a few months out. Unfortunately it is a much more significant problem to solve than it was for campaigns. I'll try and keep this thread posted as we come closer as we begin work on this project.

December 16, 2011 09:06 AM