Help needed: Program or configure a NEW web crawler

Hi guys,

this is an update thread to one I opened a bit over 3 years ago. the ae911truth.org website has a new design, and that has changed the way they display a list of petition signers, which I wish to read out and download into a table format from time to time.

So I need someone familiar with some easy web-crawling and XML-reading techniques who can program or customize a crawler for me to read out data records from a website.

Here is the new site:
http://www.ae911truth.org/signatures/#/AE/

This page has almost 3000 links to personal profiles; each link has a local href like this:
http://www.ae911truth.org/signatures...eLafayetteCAUS

Previously, those links refered to .txt files which contained XML, and the job was to parse the XML. Now, the content seems to hidden behind cloudflare somehow.

These petition signature records are based on a database with field names contained in the XML files. Now, I struggle to see the markup text that my browser (Chrome on Win7) parses.

What I want to get, with the help of a script that I can run on my amateur Win7 notebook with free amateur tools, is a table in a form like CSV/spreadsheet similar to this:

Code:

---

url|first_name|middle_name|last_name|title|degree|city|state|country|occupation_status|tech_biography|statement_911|photo|license_info
xml/supporters/U/KenGorskiEl-PasoTXUS.xml.txt|Ken||Gorski||B Architecture Professional Degree, University of Kansas, 1972|El Paso|TX|US|Degreed + Licensed|I'm a licensed architect and AIA member.|I am supportive of the intent for a complete investigation of the 9/11. Questionable structural and architectural explanations have heretofore been provided to the public.|6477 TX

---

(Same info must go into same column every time; I believe that all .txt files contain tags for every data item, so it would suffice to output just the data without headers, provided you return an empty field / "|" sign as field delimiter when tag contains no CDATA)

And the same then for
http://www.ae911truth.org/signatures/#/General/A/
http://www.ae911truth.org/signatures/#/General/B/
etc.


Am I making sense?

HELP!

Thx ;)

Winnie Mandela dead

Couldn't see a thread on this.

Horrible, horrible woman

Is criticism of Israel antisemitic?

A touchy topic to be sure, but an earnest question. As I understand it those who believe God gave the Jews the land of Israel are using a religious interpretation. I'm an atheist so I don't think there is a God to give land to anyone . Yet are atheists expected to recognize that claim of God given land? I personally believe that the Palestinians have been treated like 3rd class citizens. Yet to question that God gave the Jews the land of Israel ( I don't believe in God) and to feel the Palestinians should have equal rights- I have been called an anti Semite. Some Jewish critice of Israeli policy (Norman Finkelstein and Noam Chomsky) have been called anitsemites simply for criticizing the Israeli government. . I always thought the creation of Israel was an artificial construct created out of the British Mandate of Palestine at the end of WW2

When is criticism of Israel valid, and when is it antisemitism?

Youtube has had a major crash

Looks like the youtube channels are down. Actual videos play OK, but if you try to go to any user's actual channel, you get...

500 Internal Server Error

Sorry, something went wrong.

A team of highly trained monkeys has been dispatched to deal with this situation.

Do Truthers understand basic physics?

I have literally lost count how many Truthers asserted that the planes should have bounced off the towers like an empty tin can being thrown against a wall. I have lost count the number of Truthers who didn't understand the forces involved when the towers started to collapsed. They argue that collapse should have stopped midway, and a total collapse was proof controlled demolition. Physics seems to explain the collapse. Do Truthers believe in physics? They pretend to , but I'm not sure they understand it at all.

Why did the Protestant Reformation give rise to so many splinter sects?

It seems in the centuries following the Protestant Reformation, the Protestant branch of Christianity further fragmented into possibly hundreds of smaller groups ( from the Lutherans to the Calvinists to the Quakers to the Baptists to the Methodists, etc). Why are there so many sub sects? As an atheist I see it as an example of religious people being inconsistent with their own religion just making up their religion as they go along- but could there be somthing within Protestant sub group of Christianity that lead to such a massive fragmentation?

How to calculate calories burned in walking?

I google it . http://www.womanandhome.com/health-a...alking-206766/
No explanation is provided.

How to get formula based on physics from a school?

Thanks.

Sent from my SM-G935P using Tapatalk