New Civilization News: Google doesn't like me any more    
 Google doesn't like me any more16 comments
picture 12 Aug 2005 @ 23:48, by Flemming Funch

Well, Google likes some things I do A LOT. This blog has PageRank 7, which is fabulous. But another site which I'm more concerned about is my Opentopia site. That's a site that both is intended as a service, but also to be an income producing activity. It has an assortment of openly available collections of data, like Wikipedia's encyclopedia and the Open Directory. And some more unique pieces, like the gallery of Web Cams found in Google.

Recently the site had started to look rather promising. In terms of money, that means that a lot of people come by and click on the ads. The last two months to the tune of a little more than $30 per day, or $1000 per month. That is not huge, but it is big enough to imagine I could make a living from it, if I made it better and more people came by. And the ads that the money comes from are all from Google AdSense, as that just happens to be what works best, and what most people are comfortable with.

Lots of sites link to some part of Opentopia. 3-4000. So, people are coming in from those. But the majority of people come from Google itself, from having searched on one thing or another, which exists on the site. Google had indexed a great deal of the site, so there were many entries, and the main pages got a good PageRank as well.

But, suddenly, on the first of August, Opentopia disappeared without a trace on Google. Well, not entirely without a trace. I invented the word, and it didn't exist at all a few months ago. And, today, 49,200 webpages mention the word. But, what disappeared was everything at all in Google's index that is for the Opentopia site itself. If you ask for any pages on the site, you get:
Your search - site:opentopia.com - did not match any documents.

So, zippo, zilch, nothing. Not even the home page. The site doesn't exist as far as Google is concerned. Never heard of it. No pages there. No search results. No traffic.

And, instantly, my traffic dropped, as did my Google AdSense income, to around $5 per day.

Now, normally Google is my friend. I think Google is a great company. But if they basically own the majority of the web, it is also a cause for alarm. Getting dropped from Google is a bit like having your ID card revoked by the government. You don't exist. Google entries and Google PageRank is to a large degree a currency. Something you invest in and use and spend. But your account might be emptied over night, and there's no bank teller you can go and talk to.

See, Google's operation is so huge that there isn't exactly anybody home to talk to about this kind of thing. They of course can't answer everybody's personal questions about 10 billion webpages. But how about MY website? Last month it was in the top 30,000 sites on the net in terms of traffic. That's not super-elite, but it does make me somebody a little bit. But that doesn't really make much difference.

If one has an issue with Google, there is a support form one fills in. It doesn't really matter what one fills in - one gets an automatic reply back, which refers to their FAQs, explaining the basic rules for how one needs to behave if one wants to be listed in Google. I've read all of that many times, and I think I'm following all the rules. But one of Google's algorithms must think otherwise. Mind you, one doesn't get any kind of indication about what exactly might have gotten one's site banned. Anyway, the next step is that one then writes to them again, pleading for one's case, hoping that some real person might answer, and then look at it. That might or might not happen. Depends a bit on what one writes. The best advice seems to be to write a brief message which explains that maybe one might accidentally have done something bad in the past, and one has taken steps to clean that up, and would they please, please look at the site again.

The problem is that I don't know exactly what I might have done wrong. Quite possibly nothing. But there's a lot of pages on Opentopia. More than 1 million. So it is entirely possible that Google's spider has concluded that it just goes on forever, and that it is some kind of trick, or a site filled with random junk, to attract search engine traffic.

I have indeed noticed more and more sites like that recently. Sites that include random excerpts from random other webpages. Obviously generated by some program, and obviously to get listed in search engines. And some of them have probably succeeded well. So my guess is that Google has changed something, to try to clamp down on sites with large numbers of phoney pages.

And, well, I have a lot of pages that aren't terribly original. Copies of Wikipedia and DMOZ. None of those folks have anything against other sites mirroring their content, and, for that matter, they make it relatively easy to do, by providing regular database dumps. But how does Google's spider know the difference between a mirror of Open Directory and a random content generator? I don't know. That's probably not an easy problem to solve.

One thing that might make a difference is Google SiteMaps. Essentially one generates a map of one's site and tells Google about it, and it might help their spider do a better job at indexing it. I haven't used that feature before. I'm thinking it might be helpful if they know exactly how many pages there are, so they don't think they're just beeing tricked by some auto-generated content.

Well, I don't really know. I thought I was pretty knowledgable about this kind of thing. But now I'm an outcast, pressing my nose against the window, trying to get a glimpse of what everybody else is doing inside in the warm Google livingroom.

Well, luckily I can write about it here on my Big Ass PageRank 7 WebLog. But it still isn't fair.


[< Back] [New Civilization News]

Category:  

16 comments

13 Aug 2005 @ 02:55 by ming : Google
Google's ways can be inscrutable, even though lots of people are talking about it and sharing what they know. Can take months and months before they bother to come visiting the site the first time. After having watched the logs carefully on a number of sites, I thought I knew the rhythm. Well, I sort of do. I just don't know yet what it is one gets kicked off for.  

13 Aug 2005 @ 04:47 by UnGoogle @68.110.79.93 : Circumvent Google for Income...
You might want to consider "pay-to-Surf" programs for income and circumvent the "SEO" thing altogether... I mean, the concept is super simple, doesn't take /that/ much time, and can earn you a few thousand $$$/month whilst you focus on your real passion (like NCN).

Take {link:http://www.studiotraffic.com/index.php?refid=373780|StudioTraffic} for example: Going now over 2 and a half years paying consistently 1% per day... Indications are that it may well successfully grow for another 2 and a half years.

Also, although there are many of these programs popping up all over the net, {link:http://www.paidresponse.com/register.php?refid=1020560|Paid Response} is another one on the solid rise being successful (meaning paying) consistently since April.

I mean, they may not be the most "meaningful" of persuits online, but if its cashflow you want, circumvent Google and go Pay-to-Surf!  

13 Aug 2005 @ 08:27 by fleer : perhaps
somebody at dmoz.org banned you. You only have 2 entries there and perhaps an editor saw your site as kind of unoriginal content. Like the incredible numbers of spyware serving empty sites.

Also you don´t have a kind of "about" on the page. Maybe a statement that declares opentopia a spyware free site could help getting you back in google.

But there are also rumours on the net, that M$ is buying into or collaborating with google. And that could also be a reason. Ofcourse I can´t find a link for that story right now... :/  

13 Aug 2005 @ 09:30 by ming : Bans
Well, nobody at dmoz should be able to decide much more than whether to include me in their directory or not. Which google then carries, as does my site. But I don't think anybody goes around banning anybody there.

Some more human "about" info might be an idea. Although I think this is an automated thing, rather than somebody sitting evaluating sites, to decide whether they're worthy or not.  

13 Aug 2005 @ 09:37 by fleer : found this
about google in Danish.

In short some hosting companies block search engines from indexing. Perhaps that could be the problem.

{link:http://www.seo-debat.dk/debat/Forum17/HTML/000075.html|Index af google, udbyder blokerer måske (Danish)}  

13 Aug 2005 @ 09:50 by ming : pay2surf
Well, I have tried StudioTraffic and other program like it. Well, it is certainly not the worst of that kind of programs I've seen. It is something one might try if one really has no real traffic for some new site. But I found I mostly had wasted my time. I mean, I'm looking for real visitors who are interested in something on the site, and I'm looking for 10s of thousands of them. So far, the best way I know of doing that, other than having a lot of pages listed in Google, is to invent something cool and useful that a lot of people like and link to, and pass around to their friends, and come back to. My time would be better spent doing that, rather than going around clicking on random websites. Like look at my webcam thing. Took two days to make, and it had 50,000 visitors in less than a week. Because it happened to be interesting.  

13 Aug 2005 @ 09:54 by ming : Search engines
Flemming, well, I'm running my own servers, so it isn't that. I could understand if some regular ISPs might be concerned about search engine traffic for regular accounts. Like, for opentopia there, Google would get a page a second, 24/7, for months. And a couple of other major players have been almost as busy. Sometimes a gigabyte a day, just from search engine spiders.  

13 Aug 2005 @ 10:13 by fleer : okay
check your newciv online mail :)  

13 Aug 2005 @ 14:40 by jstarrs : This made a lot of noise...
..not long ago & it seems, by what you're describing, to be right in course.
http://mccd.udc.es/orihuela/epic/ols-master.html  

13 Aug 2005 @ 15:35 by fleer : thanks jstarrs
I hadn´t seen that one.
But yes, where does a commercial driven internet go in regards to democracy ?
We should give that a careful consideration.
Because an EPIC system could possibly stop important things from being discovered.
Like e.g. echelon.  

13 Aug 2005 @ 16:00 by ming : EPIC
And the problem is that we hardly resist when the guys who're talking over the world are "the good guys" and they're cool. We welcome the nice free innovations along the way. And as long as they give us what we want, we're happy. But we might suddenly find ourselves in a world where most information is controlled by one or two companies. And even if they still have the best of intentions, their automated algorithms won't always be right. Our access to information and ability to communicate on the net is only becoming more and more crucial. So, yes, we should be worried that there's no democracy in place that safeguards it. Just some algorithms.

A bit like a government run by automatic algorithms. Some program decides you're probably up to something bad, based on some pattern matching algorithm. So, it sends out a robocop and locks you up in jail. And if you have a complaint about that, you can talk with another program, which will decide based on some algorithm whether it has any merit or not. Scary, of course.

And, well, one can think of mechanisms that might minimize such problems. Like, what makes google useful and gives it its power is that it analyzes our choices. It indexes web pages based on how popular we make them seem to be, by how much we link to them. That isn't fool proof, of course. It could be taken to another level. Our personal choices could be mined more thoroughly, in more dimensions. If an automatic decision is made, and a bunch of people think it wasn't fair, that should be part of the feedback loop, and it should be able to correct itself.

Like, back to my website, if Google was smart enough to notice that it wasn't just a matter of blind links, but a lot of peple actually liked my website, and found it useful, they could make a better decision. It is unfortunately still possible to create fake "popularity" by making bogus links between websites. But that whole thing could be improved, so that real choices better could be identified.  

13 Aug 2005 @ 16:09 by ming : Keywords
Flemming, ah, thank you for the message! Indeed it seems I have a problem with my keywords. I have a great many pages that use the same keywords, even though they have different content. Like, all my directory pages say "search directory" and that kind of thing. Which indeed might make it look like I'm trying to cheat or something. I will change that right away.  

13 Aug 2005 @ 16:54 by jstarrs : Recently...
..there was a wave of panic from protectionist US groups about 'the UN wanting to control internet' when, in fact, it was the US doing a double take on it's decison to open up ICANN (which is US controlled) to a more international control, thus reflecting the present international (of course) development of internet.
More US paranoia.
http://www.wgig.org/  

13 Aug 2005 @ 17:52 by ming : Keywords Again
Actually it is kind of embarrassing. I've done some really obvious SEO 101 blunders without noticing. You know, like a half million pages with the same title, keywords and description. That's a no-no. Plus it would make the pages show badly in the listings, because it isn't clear what they're about. And not like I didn't know, I just didn't notice. Anyway, now I've corrected that, and I bet that will make a difference. So, thank you, Flemming, for pointing it out!  

13 Aug 2005 @ 19:03 by fleer : glad to help out
Flemming. I hope google will see the difference soon. There´s still a lot of sites that point to opentopia. I suggest that you try submitting via open directory or just wait it out. There used to be some place where you could seek a kind of autounban for sites that somehow was dropped (4 years back). But perhaps it´s been removed, because of massive requests. Also some 434 on del.icio.us have the webcam page on their bookmarks page. So it´s got some good pageranking in favor of it. Also try to submit it to some well established directories. If it´s not already listed. I don´t think it´s in yahoo.com, so that would be a primer :) geeknerd ;) - must say that after starting using a CMS system I got a notion of the extent of your programming skills. Awesome !

jstarrs;
I think the reason for americans being so much against UN goes way back to when Martin Luther King, jr was active in the southern parts of US with the freedom marches (- think they were called that). The UN flag was brought along on some of the marches. Although many have forgotten that. Not saying that they´re racists or anything. They just don´t remember how it started and what the reason for UN resentment was in the first place.  

13 Aug 2005 @ 19:35 by ming : Directories
I've submitted to Open Directory several times, but so far they haven't reacted. Yeah, that's an important one. As far as I remember, Yahoo is using that too. And, yes, a considerable number of sites link to opentopia. And, just this month so far, 23,000 people came by from a recent link on stumbledupon.com, and about 9,000 from digg.com. And its been on BoingBoing and Slashdot and stuff like that. The main page had PageRank 7, as far as I remember, and several of the other pages had 6, so it isn't bad at all. As long as Google starts listing it again. Things mostly get interesting when it gets past 100,000 unique visitors in a month. The two months before this one were 120,000 and 160,000. Without Google I'll have something like 50-60,000 visitors this month. Which isn't bad by any standard. But, incidentally, the people who come from Google listings also click a lot more on the Google ads than either frequent visitors or the people who come from sites that list the latest cool stuff. Because they actually are looking for something, and the context sensitive ads become an added service, rather than the minor annoyance it is to everybody else.  

Other entries in
8 Jul 2010 @ 02:27: Truth: superconductivity for scalable networks
11 Mar 2010 @ 17:55: The CascoDuro Case and who laughs last..
27 May 2008 @ 14:45: Thoughts on Navigating the Paradigm Shift
19 May 2008 @ 14:49: Mind Enslavement
15 May 2008 @ 06:22: 1001 Dialogues - 100 001 Actions for Dialogues and Unity in Diversity
20 Apr 2008 @ 10:57: How we unconsciously resist becoming enlightened, and what to do about it!
8 Apr 2008 @ 06:31: An open Dialogue on the Nature of Reality.
22 Feb 2008 @ 16:36: Blogging or Logging
9 Jan 2008 @ 22:45: A Communication Model
26 Oct 2007 @ 08:09: Humanities new “Canon” ?? - a new German Bildungskanon ??


[< Back] [New Civilization News] [PermaLink]?