Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive
 

This website unfortunately got a larger number of dead links when it was restructured. Seach engines don't like this. That's why I wrote a go program which helped me to locate the pages with dead links and also creates a sitemap.

Following pages are reported:

1) Internal pagelinks which are not OK (404, 403, ...)

2) External pagelinks which are not OK (404, 403, ...)

3) External pagelinks used on the website which are OK (200)

 

and in addition every page which refers to the page. It's boring to fix this but finally I was able to remove these invalid/dead links.

 

If you want to use it - either just to create a sitemap or to detect invalid links on a website call invoke the cralwer on a Linux x86 or Raspberry inthe following way. There is no go installation required.

 

curl https://raw.githubusercontent.com/framps/golang_tutorial/master/genSitemap/startCrawler.sh | bash -s -- https://<website>>

 

Now you can restart the crawler every time with ./startCrwaler.sh. Option -worker numberOfWorkers can be used to change the default number of workers.

Add comment

*** Note ***

Comments are welcome. But in order to reject spam posts please consider following rules:
  1. Comments with string http are rejected with message You have no rights to use this tag
  2. All comments are reviewed by hand and thus it usually takes one day until a comment will be published.