It is currently Tue Oct 23, 2018 10:26 am




Post new topic Reply to topic  [ 6 posts ] 
 Short Guide To Robots.txt For SEO 
Author Message
Moderator
Moderator
User avatar

Joined: Sun Sep 03, 2006 5:00 pm
Posts: 4263
Location: England
Post Short Guide To Robots.txt For SEO
What is it?
robots.txt is a regular text file that sit's in the public_html folder of your website.

What does it do?
Basically it tells the bots information, in this case it tells search engines which of them are allowed to crawl your site and which are not, it also states which files to index and which to leave alone. Robots.txt does hold more information but for now i will just use the above examples.

Why would I be intrested?
Well if you're trying to climb the search engines this helps and you can also use robots.txt to save your bandwidth by stopping search engines going to pages that you consider unecessary.

Is this compulsory for SEO?

Truthfully, I don't think it is compulsory for SEO as you can gain a SE presence without the use of a robots.txt file but it does no harm having it and it helps the SE's out with their indexing etc'

Right, how do I do this?
Google the words "robots.txt" and you should find a website that creates the file for you, it takes no time at all. once you have the file upload it to your public_html folder. I got mine from www.mcanerin.com/EN/search-engine/robots-txt.asp

I don't have a website!
Build one

Ok, I read the information above but i'm stuck!
Leave a comment below or PM me and i'll help you out. I really is as easy as 1,2,3.

What does this look like on my site?

example: www.greatlumley.com/robots.txt 99% of people will never see it, as stated above it's for SE's.

Note: robots.txt will give instructions to search engines. Spambots ignore robots.txt or choose not to follow instructions so this will not stop spambots crawling your website.

:)

_________________
Help Support PowerSellersForum By Using Our ebay bargain finder.

SaleHoo Wholesale - A Specialist Wholesale Directory.

Build A Niche Store Today (Click Here)


Sun Dec 30, 2007 12:46 pm
Profile WWW
member
member

Joined: Sun Sep 30, 2007 9:09 pm
Posts: 39
Location: Essex
Post 
Hi Billy,

Thanks for sharing that info with us, i've just made it up for my site, its really easy :D

http://www.bideasynow.com/robots.txt

Not sure about the permissions though at the mo its 0644 is that ok or should it be 0755 ???

Thanks

Gerry


Sun Dec 30, 2007 4:06 pm
Profile WWW
Moderator
Moderator
User avatar

Joined: Sun Sep 03, 2006 5:00 pm
Posts: 4263
Location: England
Post 
Bideasynow wrote:
Hi Billy,

Thanks for sharing that info with us, i've just made it up for my site, its really easy :D

http://www.bideasynow.com/robots.txt

Not sure about the permissions though at the mo its 0644 is that ok or should it be 0755 ???

Thanks

Gerry


I'm not 100% but I think it will be fine as I think 0755 is to edit the file and search engines only need read the file not edit it, I may be wrong though, someone please correct me if I am.

_________________
Help Support PowerSellersForum By Using Our ebay bargain finder.

SaleHoo Wholesale - A Specialist Wholesale Directory.

Build A Niche Store Today (Click Here)


Sun Dec 30, 2007 5:02 pm
Profile WWW
member
member

Joined: Fri Dec 18, 2009 8:49 pm
Posts: 9
Location: West Yorkshire
Post Re: Short Guide To Robots.txt For SEO
It is worth bearing in mind that you don't need to do anything if you want the search engine spiders to crawl your entire website - they will do this irrespective of whether there is a robots.txt file attached.

Their main value comes from preventing them from indexing and following links on pages, i.e. when you see noindex,nofollow in the page source :D

Anthony

_________________
Car Boot Sale Advice - http://www.everything-car-boot.co.uk


Fri Dec 18, 2009 9:08 pm
Profile WWW
member
member

Joined: Tue Jul 31, 2018 4:12 am
Posts: 39868
Post Re: Short Guide To Robots.txt For SEO
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860
bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860bil860


Wed Aug 29, 2018 6:39 am
Profile
member
member

Joined: Tue Jul 31, 2018 4:12 am
Posts: 39868
Post Re: Short Guide To Robots.txt For SEO
http://audiobookkeeper.ruhttp://cottagenet.ruhttp://eyesvision.ruhttp://eyesvisions.comhttp://kinozones.ruhttp://laserlens.ruhttp://medinfobooks.ruhttp://mp3lists.ruhttp://spicetrade.ruhttp://spysale.ruhttp://stungun.ruhttp://largeheart.ruhttp://lasercalibration.ruhttp://laserpulse.ruhttp://laterevent.ruhttp://latrinesergeant.ruhttp://layabout.ruhttp://leadcoating.ruhttp://leadingfirm.ruhttp://learningcurve.ruhttp://leaveword.ruhttp://machinesensible.ruhttp://magneticequator.ruhttp://magnetotelluricfield.ruhttp://mailinghouse.ru
http://majorconcern.ruhttp://mammasdarling.ruhttp://managerialstaff.ruhttp://manipulatinghand.ruhttp://manualchoke.ruhttp://nameresolution.ruhttp://naphtheneseries.ruhttp://narrowmouthed.ruhttp://nationalcensus.ruhttp://naturalfunctor.ruhttp://navelseed.ruhttp://neatplaster.ruhttp://necroticcaries.ruhttp://negativefibration.ruhttp://neighbouringrights.ruhttp://objectmodule.ruhttp://observationballoon.ruhttp://obstructivepatent.ruhttp://oceanmining.ruhttp://octupolephonon.ruhttp://offlinesystem.ruhttp://offsetholder.ruhttp://olibanumresinoid.ruhttp://onesticket.ruhttp://packedspheres.ru
http://pagingterminal.ruhttp://palatinebones.ruhttp://palmberry.ruhttp://papercoating.ruhttp://paraconvexgroup.ruhttp://parasolmonoplane.ruhttp://parkingbrake.ruhttp://partfamily.ruhttp://partialmajorant.ruhttp://quadrupleworm.ruhttp://qualitybooster.ruhttp://quasimoney.ruhttp://quenchedspark.ruhttp://quodrecuperet.ruhttp://rabbetledge.ruhttp://radialchaser.ruhttp://radiationestimator.ruhttp://railwaybridge.ruhttp://randomcoloration.ruhttp://rapidgrowth.ruhttp://rattlesnakemaster.ruhttp://reachthroughregion.ruhttp://readingmagnifier.ruhttp://rearchain.ruhttp://recessioncone.ru
http://recordedassignment.ruhttp://rectifiersubstation.ruhttp://redemptionvalue.ruhttp://reducingflange.ruhttp://referenceantigen.ruhttp://regeneratedprotein.ruhttp://reinvestmentplan.ruhttp://safedrilling.ruhttp://sagprofile.ruhttp://salestypelease.ruhttp://samplinginterval.ruhttp://satellitehydrology.ruhttp://scarcecommodity.ruhttp://scrapermat.ruhttp://screwingunit.ruhttp://seawaterpump.ruhttp://secondaryblock.ruhttp://secularclergy.ruhttp://seismicefficiency.ruhttp://selectivediffuser.ruhttp://semiasphalticflux.ruhttp://semifinishmachining.ruhttp://tacticaldiameter.ruhttp://tailstockcenter.ruhttp://tamecurve.ru
http://tapecorrection.ruhttp://tappingchuck.ruhttp://taskreasoning.ruhttp://technicalgrade.ruhttp://telangiectaticlipoma.ruhttp://telescopicdamper.ruhttp://temperateclimate.ruhttp://temperedmeasure.ruhttp://tenementbuilding.ruhttp://ultramaficrock.ruhttp://ultraviolettesting.ruhttp://jobstress.ruhttp://jogformation.ruhttp://jointcapsule.ruhttp://jointsealingmaterial.ruhttp://journallubricator.ruhttp://juicecatcher.ruhttp://junctionofchannels.ruhttp://justiciablehomicide.ruhttp://juxtapositiontwin.ruhttp://kaposidisease.ruhttp://keepagoodoffing.ruhttp://keepsmthinhand.ruhttp://kentishglory.ruhttp://kerbweight.ru
http://kerrrotation.ruhttp://keymanassurance.ruhttp://keyserum.ruhttp://kickplate.ruhttp://killthefattedcalf.ruhttp://kilowattsecond.ruhttp://kingweakfish.ruhttp://kleinbottle.ruhttp://kneejoint.ruhttp://knifesethouse.ruhttp://knockonatom.ruhttp://knowledgestate.ruhttp://kondoferromagnet.ruhttp://labeledgraph.ruhttp://laborracket.ruhttp://labourearnings.ruhttp://labourleasing.ruhttp://laburnumtree.ruhttp://lacingcourse.ruhttp://lacrimalpoint.ruhttp://lactogenicfactor.ruhttp://lacunarycoefficient.ruhttp://ladletreatediron.ruhttp://laggingload.ruhttp://laissezaller.ru
http://lambdatransition.ruhttp://laminatedmaterial.ruhttp://lammasshoot.ruhttp://lamphouse.ruhttp://lancecorporal.ruhttp://lancingdie.ruhttp://landingdoor.ruhttp://landmarksensor.ruhttp://landreform.ruhttp://landuseratio.ruhttp://languagelaboratory.ruhttp://factoringfee.ruhttp://filmzones.ruhttp://gadwall.ruhttp://gaffertape.ruhttp://gageboard.ruhttp://gagrule.ruhttp://gallduct.ruhttp://galvanometric.ruhttp://gangforeman.ruhttp://gangwayplatform.ruhttp://garbagechute.ruhttp://gardeningleave.ruhttp://gascautery.ruhttp://gashbucket.ru
http://gasreturn.ruhttp://gatedsweep.ruhttp://gaugemodel.ruhttp://gaussianfilter.ruhttp://gearpitchdiameter.ruhttp://geartreating.ruhttp://generalizedanalysis.ruhttp://generalprovisions.ruhttp://geophysicalprobe.ruhttp://geriatricnurse.ruhttp://getintoaflap.ruhttp://getthebounce.ruhttp://habeascorpus.ruhttp://habituate.ruhttp://hackedbolt.ruhttp://hackworker.ruhttp://hadronicannihilation.ruhttp://haemagglutinin.ruhttp://hailsquall.ruhttp://hairysphere.ruhttp://halforderfringe.ruhttp://halfsiblings.ruhttp://hallofresidence.ruhttp://haltstate.ruhttp://handcoding.ru
http://handportedhead.ruhttp://handradar.ruhttp://handsfreetelephone.ruhttp://hangonpart.ruhttp://haphazardwinding.ruhttp://hardalloyteeth.ruhttp://hardasiron.ruhttp://hardenedconcrete.ruhttp://harmonicinteraction.ruhttp://hartlaubgoose.ruhttp://hatchholddown.ruhttp://haveafinetime.ruhttp://hazardousatmosphere.ruhttp://headregulator.ruhttp://heartofgold.ruhttp://heatageingresistance.ruhttp://heatinggas.ruhttp://heavydutymetalcutting.ruhttp://jacketedwall.ruhttp://japanesecedar.ruhttp://jibtypecrane.ruhttp://jobabandonment.ru


Thu Sep 06, 2018 2:06 am
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Web Design Wigan Dog Kennels and Runs