Moz Knowledge Base/Feature Requests/Moz Feature Requests

PlannedDoneNot planned

Starting A New Crawl Manually

MeganSingley
suggested this on February 24, 2011 05:41 PM

Hi. Wondering if there's a way to get around the normally scheduled weekly crawl for each of my campaigns and start a new crawl manually. For example, we are eliminating bad URLs based on the last crawl report generated for Deadline.com but would like to see how this modification has affected things by re-running a crawl. I do not see an option for this. Please advise. Thanks.

 

Comments latest first

User photo
jon.white
SEOmoz

Hey guys,

Just checking in!

Right now we don't plan to offer manual crawl in Moz Analytics in the near future. however you can use the Crawl Test  (http://moz.com/researchtools/crawl-test) to check the issues, and when they are fixed, you will see them in your next weekly crawl.

Thanks!

Jon

January 10, 2014 12:42 PM
User photo
ldachis

When we receive errors and correct them, it would be nice to recrawl immediately to ensure they were done properly rather than wait a week.  As much as I hate to suggest it, charge a little extra for each manual crawl and turn this into a revenue source...

 

February 21, 2013 07:07 AM
User photo
Connex Social

Thank you Adam for the information! My Scenario is similar in that the validation of previous errors would be great, as most errors get fixed within 24 hours of the report. We then have to wait a week for the verification which typically turns up new errors (on new content/changes/or issues which we didn't fix well enough the first time/issues that the first fix created/etc)  

 

For example our latest crawl identified that the pagination on our cms was not working well (generating duplicate content) now that issue is fixed I'd like to see if Rogerbot picks up more problems now that the content on page 2 (et al) is displaying correctly. However I may have duplicate page titles or some other issues deeper in the pagination of our content. I won't know this for a week :(

April 11, 2012 01:17 PM
User photo
Adam Feldstein
SEOmoz

Thanks for your thoughts and scenarios! i just wanted to let you know that we are thinking along the lines of creating a validation crawl that you could run at least once a week and would check to see if errors/warnings that were surfaced in the most recent crawl were fixed. This validation could run a lot faster and provide feedback about the items that you have been working on between crawls.

This would not solve cases where a crawl fails because there are sporadic issues with the target site that make it uncrawlable. However, we  just made an update to the crawler so that it will automatically retry the crawl  if the number of pages found in the crawl is less than 10. So this should help for most of such cases.

March 28, 2012 12:44 PM
User photo
davidrevis

We are trialling the product and due to an error from our web company managed to block google from crawling our site yesterday, just as SEOMOZ was recrawling our site. Result is that we only have 4 pages indexed this week instead of the 840 pages we had on 1st crawl last week. We have fixed the errors on the site and have managed to get google to recrawl the site but cant find a button to get SEOMOZ to do it. At least our crawl errors have gone down this week though!!! :)

February 17, 2012 09:38 AM
User photo
Dave Davis

For us, manual is not needed, but twice a week would be perfect.

January 12, 2012 11:14 AM
User photo
Epica

Great to hear it's on the radar - I know it's a much needed feature for myself and quite possibily others as well.

 

Ideally, I'd like to have the option to do both. 

January 03, 2012 09:33 PM
User photo
Adam Feldstein
SEOmoz

It's not possible yet, though we are looking into updating our infrastructure to support this in the future. Unfortunately it will be some time before this is in place. I expect it may take until spring or summer before we have a chance to get this in place. :-/

If you don't mind, it would be great to understand how often you need to run manual checks on your rankings. Do you need to check specific ones sporadically, or are you looking to refresh all of the rankings frequently?

Thanks again for sharing the suggestion!

Adam

December 20, 2011 09:59 PM
User photo
Epica

Great tool, thanks for sharing. However, I'd like a tool just like this that allows you to check rankings on the keywords in your campaign manually. Possible?

December 20, 2011 03:33 PM
User photo
Jackie Immel
SEOmoz

Great suggestion! If you would like to start a manual crawl, go to the Research Tools (www.seomoz.org/tools) and click "Crawl Test." This allows you to manually start a crawl for up to two sites every 24-hours. Just enter a subdomain, and you'll receive a new report within a few days. I hope this helps :)

December 20, 2011 02:29 PM
User photo
Epica

This is a much needed feature - thanks for suggesting Megan. I'd also like to manually run a position crawl for my campaigns at any time as well (at least an 'allotment' of this feature each month, IE 5 manual position checks per month, etc.)

December 16, 2011 08:08 AM