[ranjani]

I want to search & capture RSS feeds automatically and dynamically based on a certain criteria and feed it periodically into my database. What is the best programming language used for this process. I do not want to purchase an RSS aggragator, but I want to try to code myself with the help of some source code.

I have read the reply to one of the related posts in this forum, which provided the following information:
-------------------------------------
So long as you want to code your own the tools are there: CPAN has the rss modules:

http://search.cpan.org/search?query=rss&mode=all
http://search.cpan.org/~autrijus/XM...SS/Aggregate.pm

Build a spider to fetch your rss feeds and use the concepts described here to index:

http://www.perl.com/pub/a/2003/09/25/searching.html
http://www.perl.com/pub/a/2003/02/19/engine.html

Then all you need to do is build a front end to it all. All in all, not a trivial task but definitely doable.
-----------------------------------

Concerning this reply, I have a few questions.

- For a novice programmer of RSS feed search, how do I integrate all links mentions in the reply above. What are the important steps to focus on when coding a RSS aggregator from scratch.

- I need some more information on how to buld a spider and what it means.

- Is there any open-source code which performs search based on RSS feeds.

- Since the database is programmed using MySql, how do I intergrate another programming language.

- How to search for RSS feed automatically, instead of subscribing to each website individually or entering url manually.

- How do i run the chunks of code i get in a composed manner ?

All suggestions and guidance would be very helpful and deeply appreciated.

Thank you,
Ranjani.

[Chas]

Hi Ranjani,

This looks like a reply to your original post I made a while back. Let me see if I can answer some of your questions below:

Originally Posted by ranjani

I want to search & capture RSS feeds automatically and dynamically based on a certain criteria and feed it periodically into my database.

Can you be a bit more specific as to what "dynamically based on a certain criteria" and "feed it periodically" mean? With generalities like it's impossible from someone to answer your questions as specific as you want.

Originally Posted by ranjani

What is the best programming language used for this process.

Perl, of course

Originally Posted by ranjani

I do not want to purchase an RSS aggragator, but I want to try to code myself with the help of some source code.

Again, as I stated in my original reply, this is no trivial task. I'd be looking at a few weeks worth of work easy with my schedule. But, I'm with ya, I'd code it myself too.

[snip /]

Originally Posted by ranjani

- For a novice programmer of RSS feed search, how do I integrate all links mentions in the reply above. What are the important steps to focus on when coding a RSS aggregator from scratch.

This is standard Perl, albeit not basic. The best I can do is point you to a few good books aside from coding it for you. See response above...

Originally Posted by ranjani

- I need some more information on how to buld a spider and what it means.

Search perl.com for starters; there was a series of articles on spidering a while back. Also, check google. There is tons of info on this out there. You may even want to check search.cpan.org. There is most like some base class on there that would get you going.

Originally Posted by ranjani

- Is there any open-source code which performs search based on RSS feeds.

I would be very surprised if there was not something open source out there. See answer to previous question for more details.

Originally Posted by ranjani

- Since the database is programmed using MySql, how do I intergrate another programming language.

The DBI and DBD::mysql modules are what you're after. They will provide an interface to your MySQL database in Perl.

Originally Posted by ranjani

- How to search for RSS feed automatically, instead of subscribing to each website individually or entering url manually.

That's where your spider comes in. Although you will need to feed it something to get it started.

Originally Posted by ranjani

- How do i run the chunks of code i get in a composed manner ?

Can you elaborate please? I'm not sure what you're asking here.

~Charlie