Browse Source

Blocks some bots from crawling

Abhinav Sarkar 10 months ago
parent
commit
c486fe2ca4
1 changed files with 6 additions and 0 deletions
  1. 6
    0
      robots.txt

+ 6
- 0
robots.txt View File

@@ -1,3 +1,9 @@
1 1
 Sitemap: https://abhinavsarkar.net/sitemap.xml
2 2
 User-agent: *
3 3
 Disallow: /drafts/
4
+
5
+User-agent: SemrushBot
6
+Disallow: /
7
+
8
+User-agent: Barkrowler
9
+Disallow: /

Loading…
Cancel
Save