Azfeeds proxied
Azfeeds just got a nice proxy option. Too bad the database did not get any faster by this, but a round robin DNS helps with some speed issues.
Changes: I implemented a little script that seeds out old news into the archive, keeping only 3 weeks of news in the main database. That uped the speed significantly and speeded up the insertion process, so one cycle of fetch can run a lot faster.
Problems: this whole change process affected the server for a few days (last weekend), however custom feeds weren't bothered at all (they are premade). Database performance was terrible and sometimes you could not make a query for 2 hours. This is an experimental project so sometimes we have to experiment.... sorry.
Now everything seems to be working fine and we can start adding a site submit feature soon. We also have to make a living and this site is not a profit oriented project, so it might take a while to add that feature.
New server new speed, custom feeds....
Our old FreeBSD box has been obsoleted by the size of the database. So we had to move on to a new server. Now queries execute 10-50 times faster. OK I know I will still have to optimise and restrict because it is just plain horror to allow anyone to executer real-time full text searches on 1 million records of several text fields. Just not right:).
Anyway here is the change:
OLD server : FBSD, 2G Celeron, 512M ram, ATA,
ISP: ColoDallas
NEW server: Linux, 3G HT P4, 1G ram, SATAISP: Layered Tech
Can you beleive it is 10 times faster on FT searches? Well you can test the old DB here: this article has been changed, the old server is not accessible anymore.
By the way, to reduce loads, that server will be used as a proxy to our site... I am still thinking of how I will do it, but if Azfeeds finally rolls out, we will need at least 1 mirror/proxy and an army of news fetching clients .... well in the works ... but as many others, it is an experimental project so time is limited.
The decision came as I tried to test our new custom feeds feature (you will be eble to create a filter and we generate a feed for you) and our old server was so slow, I kept waiting for 1 minutes between search tests. I hate to wait for a computer :) so I moved the whole thing immediately.
By the way custom feeds: soon it will be ready, and I tell you, you will LOVE it :).